Jun 15 05:46:18.870364 ip-10-0-128-226 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jun 15 05:46:18.870376 ip-10-0-128-226 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jun 15 05:46:18.870384 ip-10-0-128-226 systemd[1]: kubelet.service: Failed with result 'resources'. Jun 15 05:46:18.870632 ip-10-0-128-226 systemd[1]: Failed to start Kubernetes Kubelet. Jun 15 05:46:29.119842 ip-10-0-128-226 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jun 15 05:46:29.119860 ip-10-0-128-226 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 30b17e160e104fa0887dc4d19b6c8121 -- Jun 15 05:48:41.545863 ip-10-0-128-226 systemd[1]: Starting Kubernetes Kubelet... Jun 15 05:48:41.900913 ip-10-0-128-226 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 15 05:48:41.900913 ip-10-0-128-226 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jun 15 05:48:41.900913 ip-10-0-128-226 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 15 05:48:41.900913 ip-10-0-128-226 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jun 15 05:48:41.900913 ip-10-0-128-226 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jun 15 05:48:41.903064 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.902976 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jun 15 05:48:41.907851 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907831 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 15 05:48:41.907851 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907851 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907855 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907859 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907862 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907865 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907868 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907870 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907873 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907876 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907878 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907881 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907884 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907887 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907889 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907892 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907895 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907899 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907902 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907911 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907915 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 15 05:48:41.907921 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907918 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907920 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907924 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907926 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907929 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907932 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907935 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907937 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907940 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907943 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907945 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907948 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907951 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907955 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907957 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907960 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907963 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907966 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907969 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 15 05:48:41.908439 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907971 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907974 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907977 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907980 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907983 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907985 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907988 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907990 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907993 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907996 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.907999 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908002 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908004 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908007 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908009 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908012 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908015 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908018 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908020 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 15 05:48:41.908923 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908023 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908025 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908028 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908030 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908033 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908035 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908038 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908040 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908043 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908046 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908049 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908051 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908053 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908056 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908059 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908061 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908064 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908066 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908069 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908071 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 15 05:48:41.909385 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908075 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908078 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908080 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908083 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908085 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908088 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908090 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908093 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908096 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908098 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908101 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908104 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908107 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908109 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908112 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908115 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908118 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908120 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908123 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 15 05:48:41.909897 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908127 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908839 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908847 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908850 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908853 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908856 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908859 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908862 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908865 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908867 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908870 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908873 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908876 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908879 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908882 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908885 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908887 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908890 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908892 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908895 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 15 05:48:41.910365 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908897 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908901 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908904 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908906 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908909 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908912 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908914 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908919 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908923 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908926 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908929 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908932 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908934 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908938 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908941 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908944 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908946 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908949 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908951 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908955 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 15 05:48:41.910887 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908957 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908960 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908962 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908965 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908968 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908970 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908973 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908976 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908980 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908990 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908995 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.908998 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909001 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909004 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909008 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909012 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909015 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909018 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909021 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 15 05:48:41.911386 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909024 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909027 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909030 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909033 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909035 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909038 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909041 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909043 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909046 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909048 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909051 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909053 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909057 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909061 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909063 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909066 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909069 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909071 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909074 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909076 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 15 05:48:41.911950 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909079 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909081 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909084 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909086 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909089 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909091 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909094 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909097 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909099 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909102 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909104 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909107 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909109 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909112 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909114 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909117 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909120 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909123 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909125 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909128 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 15 05:48:41.912452 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909130 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909515 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909521 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909523 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909526 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909529 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909532 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909535 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909550 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909553 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909557 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909559 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909562 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909565 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909567 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909571 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909574 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909576 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909579 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909582 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 15 05:48:41.912960 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909584 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909587 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909590 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909593 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909595 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909598 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909600 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909603 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909605 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909608 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909611 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909614 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909616 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909619 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909621 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909624 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909626 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909629 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909632 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 15 05:48:41.913463 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909635 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909639 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909643 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909645 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909648 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909651 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909654 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909657 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909659 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909662 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909665 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909667 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909670 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909672 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909675 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909677 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909680 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909682 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909685 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909687 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 15 05:48:41.913995 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909690 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909693 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909695 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909698 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909700 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909703 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909706 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909708 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909711 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909713 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909718 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909721 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909724 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909727 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909730 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909733 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909735 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909738 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909740 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 15 05:48:41.914487 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909743 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909747 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909753 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909756 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909759 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909761 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909764 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909767 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909770 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909772 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909775 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909778 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909782 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909785 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909788 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909791 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909794 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909797 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909799 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909802 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 15 05:48:41.914990 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909805 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.909807 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909893 2569 flags.go:64] FLAG: --address="0.0.0.0" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909901 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909910 2569 flags.go:64] FLAG: --anonymous-auth="true" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909915 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909922 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909926 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909930 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909934 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909937 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909941 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909944 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909947 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909952 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909955 2569 flags.go:64] FLAG: --cgroup-root="" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909958 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909961 2569 flags.go:64] FLAG: --client-ca-file="" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909964 2569 flags.go:64] FLAG: --cloud-provider="external" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909967 2569 flags.go:64] FLAG: --cluster-dns="[]" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909972 2569 flags.go:64] FLAG: --cluster-domain="" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909975 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909978 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909981 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jun 15 05:48:41.915486 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909984 2569 flags.go:64] FLAG: --container-log-max-files="5" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909988 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909990 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909994 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.909997 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910000 2569 flags.go:64] FLAG: --contention-profiling="false" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910003 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910006 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910008 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910011 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910015 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910018 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910021 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910024 2569 flags.go:64] FLAG: --enable-load-reader="false" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910027 2569 flags.go:64] FLAG: --enable-server="true" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910031 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910036 2569 flags.go:64] FLAG: --event-burst="100" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910039 2569 flags.go:64] FLAG: --event-qps="50" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910042 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910045 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910047 2569 flags.go:64] FLAG: --eviction-hard="" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910051 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910059 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910062 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910065 2569 flags.go:64] FLAG: --eviction-soft="" Jun 15 05:48:41.916093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910068 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910071 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910074 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910076 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910082 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910085 2569 flags.go:64] FLAG: --fail-swap-on="true" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910089 2569 flags.go:64] FLAG: --feature-gates="" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910093 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910096 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910100 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910103 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910106 2569 flags.go:64] FLAG: --healthz-port="10248" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910109 2569 flags.go:64] FLAG: --help="false" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910112 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-128-226.ec2.internal" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910115 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910118 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910120 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910124 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910128 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910131 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910134 2569 flags.go:64] FLAG: --image-service-endpoint="" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910136 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910139 2569 flags.go:64] FLAG: --kube-api-burst="100" Jun 15 05:48:41.916745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910143 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910146 2569 flags.go:64] FLAG: --kube-api-qps="50" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910149 2569 flags.go:64] FLAG: --kube-reserved="" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910152 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910155 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910158 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910162 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910165 2569 flags.go:64] FLAG: --lock-file="" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910168 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910171 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910174 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910179 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910182 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910186 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910189 2569 flags.go:64] FLAG: --logging-format="text" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910193 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910196 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910199 2569 flags.go:64] FLAG: --manifest-url="" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910202 2569 flags.go:64] FLAG: --manifest-url-header="" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910206 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910209 2569 flags.go:64] FLAG: --max-open-files="1000000" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910213 2569 flags.go:64] FLAG: --max-pods="110" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910216 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910219 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910222 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jun 15 05:48:41.917360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910225 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910228 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910231 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910234 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910242 2569 flags.go:64] FLAG: --node-status-max-images="50" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910245 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910247 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910250 2569 flags.go:64] FLAG: --pod-cidr="" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910254 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a2e90c8ef7d783e4d635e24fb09435f5e1f6e954583512c0944f9e57bc77b52" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910259 2569 flags.go:64] FLAG: --pod-manifest-path="" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910262 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910265 2569 flags.go:64] FLAG: --pods-per-core="0" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910269 2569 flags.go:64] FLAG: --port="10250" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910273 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910276 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-060105943d2f3391d" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910279 2569 flags.go:64] FLAG: --qos-reserved="" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910282 2569 flags.go:64] FLAG: --read-only-port="10255" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910285 2569 flags.go:64] FLAG: --register-node="true" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910288 2569 flags.go:64] FLAG: --register-with-taints="" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910292 2569 flags.go:64] FLAG: --registry-burst="10" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910295 2569 flags.go:64] FLAG: --registry-qps="5" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910300 2569 flags.go:64] FLAG: --reserved-cpus="" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910303 2569 flags.go:64] FLAG: --reserved-memory="" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910307 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910310 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jun 15 05:48:41.917999 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910312 2569 flags.go:64] FLAG: --rotate-certificates="false" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910315 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910318 2569 flags.go:64] FLAG: --runonce="false" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910321 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910324 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910326 2569 flags.go:64] FLAG: --seccomp-default="false" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910329 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910332 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910335 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910338 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910341 2569 flags.go:64] FLAG: --storage-driver-password="root" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910343 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910346 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910349 2569 flags.go:64] FLAG: --storage-driver-user="root" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910352 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910355 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910358 2569 flags.go:64] FLAG: --system-cgroups="" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910361 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910367 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910370 2569 flags.go:64] FLAG: --tls-cert-file="" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910374 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910378 2569 flags.go:64] FLAG: --tls-min-version="" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910381 2569 flags.go:64] FLAG: --tls-private-key-file="" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910384 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910387 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jun 15 05:48:41.918682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910390 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910393 2569 flags.go:64] FLAG: --v="2" Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910397 2569 flags.go:64] FLAG: --version="false" Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910403 2569 flags.go:64] FLAG: --vmodule="" Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910407 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.910410 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910524 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910528 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910533 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910537 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910554 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910557 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910560 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910563 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910566 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910569 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910571 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910574 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910576 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910579 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910582 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 15 05:48:41.919287 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910584 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910587 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910589 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910593 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910595 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910598 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910602 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910606 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910609 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910612 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910616 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910618 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910622 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910625 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910629 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910632 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910634 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910637 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910640 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910643 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 15 05:48:41.919829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910645 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910648 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910651 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910653 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910656 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910658 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910661 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910663 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910666 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910669 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910671 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910674 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910676 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910679 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910681 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910684 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910687 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910689 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910694 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910696 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 15 05:48:41.920325 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910698 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910701 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910703 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910706 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910708 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910711 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910715 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910717 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910720 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910724 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910727 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910730 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910733 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910735 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910738 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910741 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910743 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910746 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910748 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 15 05:48:41.920829 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910751 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910753 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910756 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910758 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910761 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910763 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910766 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910769 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910771 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910773 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910776 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910780 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910783 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910786 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910788 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910791 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910793 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910795 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910798 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910802 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 15 05:48:41.921312 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910804 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910807 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910809 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910812 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.910815 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.911748 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.918845 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.918863 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.918884 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.918895 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918969 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918975 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918978 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918982 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918986 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 15 05:48:41.921833 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918989 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918992 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918994 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.918997 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919000 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919002 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919006 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919010 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919015 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919018 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919023 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919027 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919031 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919035 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919039 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919044 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919047 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919050 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919052 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919055 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 15 05:48:41.922216 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919058 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919061 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919063 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919066 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919070 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919073 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919076 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919079 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919081 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919084 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919087 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919089 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919092 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919095 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919097 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919100 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919102 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919106 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919110 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919112 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 15 05:48:41.922728 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919115 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919118 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919120 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919123 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919125 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919128 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919130 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919133 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919136 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919138 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919141 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919143 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919146 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919149 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919152 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919154 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919158 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919161 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919164 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919166 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 15 05:48:41.923222 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919169 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919173 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919176 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919180 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919182 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919185 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919188 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919190 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919193 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919196 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919199 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919202 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919204 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919207 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919209 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919212 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919214 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919217 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919219 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 15 05:48:41.923932 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919222 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919226 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919229 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919232 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919235 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919237 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919240 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919243 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919245 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919248 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919251 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919254 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919257 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919259 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 15 05:48:41.924691 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919262 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.919267 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919401 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919407 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919411 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919413 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919416 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919419 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919422 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919424 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919427 2569 feature_gate.go:328] unrecognized feature gate: Example Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919430 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919432 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919435 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919437 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jun 15 05:48:41.925077 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919440 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919442 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919445 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919448 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919451 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919454 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919456 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919459 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919462 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919464 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919467 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919469 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919472 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919475 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919479 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919484 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919487 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919490 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919492 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jun 15 05:48:41.925454 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919495 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919498 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919500 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919503 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919505 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919508 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919510 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919513 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919516 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919518 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919521 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919523 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919526 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919528 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919531 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919533 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919536 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919551 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919555 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919558 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jun 15 05:48:41.925945 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919560 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919563 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919566 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919568 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919571 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919574 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919576 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919579 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919581 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919589 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919592 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919595 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919597 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919600 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919604 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919607 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919610 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919613 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919616 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jun 15 05:48:41.926419 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919620 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919624 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919627 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919629 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919632 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919634 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919637 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919640 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919642 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919645 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919647 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919650 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919653 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919655 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919658 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919660 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919663 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919666 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919669 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919672 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jun 15 05:48:41.926904 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919674 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919677 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919680 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919683 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919685 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919688 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919690 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:41.919693 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.919698 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.919836 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.921898 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.924170 2569 server.go:1019] "Starting client certificate rotation" Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.924437 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 15 05:48:41.927400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.925107 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jun 15 05:48:41.927753 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.927005 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 15 05:48:41.928168 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.928150 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jun 15 05:48:41.938505 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.938484 2569 log.go:25] "Validated CRI v1 runtime API" Jun 15 05:48:41.941845 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.941822 2569 log.go:25] "Validated CRI v1 image API" Jun 15 05:48:41.943202 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.943185 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jun 15 05:48:41.945388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.945367 2569 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a1e38b91-339e-4994-aaf5-02c34c8da0c7:/dev/nvme0n1p4 df5ba415-24df-4c5f-a51d-ce57bf62317a:/dev/nvme0n1p3] Jun 15 05:48:41.945453 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.945386 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jun 15 05:48:41.950635 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.950506 2569 manager.go:216] Machine: {Timestamp:2026-06-15 05:48:41.949083885 +0000 UTC m=+0.294287017 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109811 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b91c698b2b4729b36629e6190e39a SystemUUID:ec2b91c6-98b2-b472-9b36-629e6190e39a BootID:30b17e16-0e10-4fa0-887d-c4d19b6c8121 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1a:d1:56:ee:ab Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1a:d1:56:ee:ab Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:50:22:75:6d:04 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jun 15 05:48:41.950635 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.950629 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jun 15 05:48:41.950739 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.950716 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.120.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260608-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jun 15 05:48:41.953414 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.953383 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jun 15 05:48:41.953569 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.953416 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jun 15 05:48:41.954117 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.954107 2569 topology_manager.go:138] "Creating topology manager with none policy" Jun 15 05:48:41.954147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.954120 2569 container_manager_linux.go:309] "Creating device plugin manager" Jun 15 05:48:41.954147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.954127 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 15 05:48:41.955224 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.955213 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jun 15 05:48:41.956434 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.956423 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jun 15 05:48:41.959093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.959081 2569 state_mem.go:36] "Initialized new in-memory state store" Jun 15 05:48:41.959377 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.959367 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jun 15 05:48:41.960282 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.960272 2569 kubelet.go:486] "Attempting to sync node with API server" Jun 15 05:48:41.960324 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.960293 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jun 15 05:48:41.960324 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.960308 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jun 15 05:48:41.960324 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.960318 2569 kubelet.go:398] "Adding apiserver pod source" Jun 15 05:48:41.960458 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.960328 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jun 15 05:48:41.960692 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.960679 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 15 05:48:41.960749 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.960705 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jun 15 05:48:41.964411 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.964390 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jun 15 05:48:41.965668 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.965648 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 15 05:48:41.965800 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.965786 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jun 15 05:48:41.965855 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.965828 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jun 15 05:48:41.966021 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966009 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jun 15 05:48:41.966082 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966027 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jun 15 05:48:41.966082 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966036 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jun 15 05:48:41.966082 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966044 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jun 15 05:48:41.966082 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966056 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jun 15 05:48:41.966082 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966064 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jun 15 05:48:41.966471 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966461 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jun 15 05:48:41.966519 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966473 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jun 15 05:48:41.966519 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966484 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jun 15 05:48:41.966519 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966493 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jun 15 05:48:41.966519 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966505 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jun 15 05:48:41.966519 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.966517 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jun 15 05:48:41.967228 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.967217 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jun 15 05:48:41.967284 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.967230 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jun 15 05:48:41.970669 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.970647 2569 server.go:1268] "Started kubelet" Jun 15 05:48:41.970759 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.970721 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jun 15 05:48:41.970824 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.970755 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jun 15 05:48:41.970871 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.970833 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jun 15 05:48:41.971670 ip-10-0-128-226 systemd[1]: Started Kubernetes Kubelet. Jun 15 05:48:41.971949 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.971915 2569 server.go:310] "Adding debug handlers to kubelet server" Jun 15 05:48:41.972554 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.972513 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jun 15 05:48:41.975996 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.975977 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jun 15 05:48:41.976133 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.976086 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jun 15 05:48:41.976178 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.976156 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jun 15 05:48:41.977489 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.977459 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jun 15 05:48:41.977686 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.977671 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jun 15 05:48:41.977761 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.977734 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jun 15 05:48:41.977761 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.977749 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:41.978396 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978381 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jun 15 05:48:41.978505 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978381 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jun 15 05:48:41.978595 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978574 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jun 15 05:48:41.978669 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978657 2569 reconstruct.go:97] "Volume reconstruction finished" Jun 15 05:48:41.978669 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978669 2569 reconciler.go:29] "Reconciler: start to sync state" Jun 15 05:48:41.978765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978679 2569 factory.go:153] Registering CRI-O factory Jun 15 05:48:41.978765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978696 2569 factory.go:223] Registration of the crio container factory successfully Jun 15 05:48:41.978765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978748 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jun 15 05:48:41.978765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978758 2569 factory.go:55] Registering systemd factory Jun 15 05:48:41.978765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978765 2569 factory.go:223] Registration of the systemd container factory successfully Jun 15 05:48:41.978959 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978785 2569 factory.go:103] Registering Raw factory Jun 15 05:48:41.978959 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.978796 2569 manager.go:1196] Started watching for new ooms in manager Jun 15 05:48:41.979180 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.979168 2569 manager.go:349] Starting recovery of all containers Jun 15 05:48:41.980889 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.980842 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jun 15 05:48:41.981819 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.981788 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jun 15 05:48:41.982255 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.980832 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-226.ec2.internal.18b92b1adc85668b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-226.ec2.internal,UID:ip-10-0-128-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-226.ec2.internal,},FirstTimestamp:2026-06-15 05:48:41.970615947 +0000 UTC m=+0.315819108,LastTimestamp:2026-06-15 05:48:41.970615947 +0000 UTC m=+0.315819108,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-226.ec2.internal,}" Jun 15 05:48:41.986400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.986242 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jun 15 05:48:41.990453 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.990429 2569 manager.go:354] Recovery completed Jun 15 05:48:41.992450 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.992423 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kfpbx" Jun 15 05:48:41.993366 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:41.993347 2569 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jun 15 05:48:41.996383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.996369 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 15 05:48:41.998297 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.998282 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientMemory" Jun 15 05:48:41.998357 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.998310 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasNoDiskPressure" Jun 15 05:48:41.998357 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.998320 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientPID" Jun 15 05:48:41.998755 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.998737 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jun 15 05:48:41.998755 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.998751 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jun 15 05:48:41.998755 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.998753 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 15 05:48:41.998897 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.998766 2569 state_mem.go:36] "Initialized new in-memory state store" Jun 15 05:48:41.999882 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.999866 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientMemory" Jun 15 05:48:41.999950 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.999891 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasNoDiskPressure" Jun 15 05:48:41.999950 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.999901 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientPID" Jun 15 05:48:41.999950 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.999921 2569 policy_none.go:49] "None policy: Start" Jun 15 05:48:41.999950 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.999934 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jun 15 05:48:41.999950 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:41.999943 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jun 15 05:48:42.000448 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.000377 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-226.ec2.internal.18b92b1ade2bc5b7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-226.ec2.internal,UID:ip-10-0-128-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-226.ec2.internal,},FirstTimestamp:2026-06-15 05:48:41.998296503 +0000 UTC m=+0.343499635,LastTimestamp:2026-06-15 05:48:41.998296503 +0000 UTC m=+0.343499635,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-226.ec2.internal,}" Jun 15 05:48:42.000803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.000792 2569 policy_none.go:47] "Start" Jun 15 05:48:42.001026 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.001010 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kfpbx" Jun 15 05:48:42.037203 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.037182 2569 manager.go:337] "Starting Device Plugin manager" Jun 15 05:48:42.037317 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.037221 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jun 15 05:48:42.037317 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.037235 2569 server.go:87] "Starting device plugin registration server" Jun 15 05:48:42.037894 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.037530 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jun 15 05:48:42.037894 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.037560 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jun 15 05:48:42.037894 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.037633 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jun 15 05:48:42.037894 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.037715 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jun 15 05:48:42.037894 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.037723 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jun 15 05:48:42.051703 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.038380 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jun 15 05:48:42.051703 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.038421 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.051703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.050264 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jun 15 05:48:42.051703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.050301 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jun 15 05:48:42.051703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.050322 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jun 15 05:48:42.051703 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.050355 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jun 15 05:48:42.052932 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.052911 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 15 05:48:42.138477 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.138438 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 15 05:48:42.140143 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.140124 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientMemory" Jun 15 05:48:42.140244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.140161 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasNoDiskPressure" Jun 15 05:48:42.140244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.140175 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientPID" Jun 15 05:48:42.140244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.140206 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.146754 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.146735 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.146848 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.146764 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-226.ec2.internal\": node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.150443 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.150425 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal"] Jun 15 05:48:42.150557 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.150480 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 15 05:48:42.151510 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.151456 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientMemory" Jun 15 05:48:42.151510 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.151489 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasNoDiskPressure" Jun 15 05:48:42.151510 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.151499 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientPID" Jun 15 05:48:42.152926 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.152909 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 15 05:48:42.153038 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.153018 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.153093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.153051 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 15 05:48:42.153701 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.153683 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientMemory" Jun 15 05:48:42.153798 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.153712 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasNoDiskPressure" Jun 15 05:48:42.153798 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.153726 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientPID" Jun 15 05:48:42.154280 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.154264 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientMemory" Jun 15 05:48:42.154361 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.154295 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasNoDiskPressure" Jun 15 05:48:42.154361 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.154308 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientPID" Jun 15 05:48:42.155571 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.155556 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.155654 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.155588 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jun 15 05:48:42.159324 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.159308 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientMemory" Jun 15 05:48:42.159383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.159338 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasNoDiskPressure" Jun 15 05:48:42.159383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.159347 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeHasSufficientPID" Jun 15 05:48:42.161715 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.161699 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.175289 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.175262 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-226.ec2.internal\" not found" node="ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.180280 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.180260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f281e2e25ec3badbb2247e6dadbdfaf5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-226.ec2.internal\" (UID: \"f281e2e25ec3badbb2247e6dadbdfaf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.180373 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.180284 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-226.ec2.internal\" not found" node="ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.180373 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.180287 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c5a5760696eae5c4e3ad3c580448608-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal\" (UID: \"1c5a5760696eae5c4e3ad3c580448608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.180373 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.180361 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5a5760696eae5c4e3ad3c580448608-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal\" (UID: \"1c5a5760696eae5c4e3ad3c580448608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.262051 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.262016 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.281300 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.281268 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f281e2e25ec3badbb2247e6dadbdfaf5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-226.ec2.internal\" (UID: \"f281e2e25ec3badbb2247e6dadbdfaf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.281300 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.281302 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c5a5760696eae5c4e3ad3c580448608-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal\" (UID: \"1c5a5760696eae5c4e3ad3c580448608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.281480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.281322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5a5760696eae5c4e3ad3c580448608-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal\" (UID: \"1c5a5760696eae5c4e3ad3c580448608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.281480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.281365 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f281e2e25ec3badbb2247e6dadbdfaf5-config\") pod \"kube-apiserver-proxy-ip-10-0-128-226.ec2.internal\" (UID: \"f281e2e25ec3badbb2247e6dadbdfaf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.281480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.281382 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1c5a5760696eae5c4e3ad3c580448608-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal\" (UID: \"1c5a5760696eae5c4e3ad3c580448608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.281480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.281385 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1c5a5760696eae5c4e3ad3c580448608-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal\" (UID: \"1c5a5760696eae5c4e3ad3c580448608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.362849 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.362815 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.463778 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.463700 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.477923 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.477896 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.482954 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.482930 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.564608 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.564571 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.665123 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.665086 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.765715 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.765626 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-226.ec2.internal\" not found" Jun 15 05:48:42.803754 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.803731 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 15 05:48:42.877829 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.877799 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.890115 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.890086 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 15 05:48:42.891745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.891730 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" Jun 15 05:48:42.909168 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.909130 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jun 15 05:48:42.923342 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.923313 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jun 15 05:48:42.923505 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.923480 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 15 05:48:42.923579 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.923521 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jun 15 05:48:42.960499 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.960467 2569 apiserver.go:52] "Watching apiserver" Jun 15 05:48:42.973640 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:42.973591 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5a5760696eae5c4e3ad3c580448608.slice/crio-3f1c0ed1a35567253a41c24598e78a62b1c2689b0c696fc93027c7140cfa29dc WatchSource:0}: Error finding container 3f1c0ed1a35567253a41c24598e78a62b1c2689b0c696fc93027c7140cfa29dc: Status 404 returned error can't find the container with id 3f1c0ed1a35567253a41c24598e78a62b1c2689b0c696fc93027c7140cfa29dc Jun 15 05:48:42.973999 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:42.973974 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf281e2e25ec3badbb2247e6dadbdfaf5.slice/crio-8aaf740011b81fadb0c9b1f2f3175465f30d2adbf99b6413046f3a40dc49df23 WatchSource:0}: Error finding container 8aaf740011b81fadb0c9b1f2f3175465f30d2adbf99b6413046f3a40dc49df23: Status 404 returned error can't find the container with id 8aaf740011b81fadb0c9b1f2f3175465f30d2adbf99b6413046f3a40dc49df23 Jun 15 05:48:42.976810 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.976789 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jun 15 05:48:42.978096 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.977634 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-pvj8k","kube-system/konnectivity-agent-6z4c5","kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t","openshift-cluster-node-tuning-operator/tuned-zzm2s","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal","openshift-multus/multus-nt2fx","openshift-multus/network-metrics-daemon-pxmpj","openshift-dns/node-resolver-zqzpb","openshift-image-registry/node-ca-g28pn","openshift-multus/multus-additional-cni-plugins-v2kln","openshift-network-diagnostics/network-check-target-8n9hm","openshift-network-operator/iptables-alerter-747dl","openshift-ovn-kubernetes/ovnkube-node-rkdkp"] Jun 15 05:48:42.978503 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.978451 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jun 15 05:48:42.979390 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.979359 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jun 15 05:48:42.979930 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.979911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:42.980032 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.980000 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:42.980381 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.980367 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 15 05:48:42.980933 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.980911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:42.981994 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.981977 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:42.983069 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.983049 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:42.983389 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.983369 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w96d2\"" Jun 15 05:48:42.983475 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.983448 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jun 15 05:48:42.983573 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.983479 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jun 15 05:48:42.984480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.984176 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jun 15 05:48:42.984480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.984210 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nt2fx" Jun 15 05:48:42.984480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.984267 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jun 15 05:48:42.984724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.984705 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-sbs9d\"" Jun 15 05:48:42.984828 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.984814 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jun 15 05:48:42.985337 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.985317 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:42.985428 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.985381 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:42.986173 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.986155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9039cf21-b4dc-4e30-bdb6-a08444213d19-kubelet-config\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:42.986252 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.986179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9039cf21-b4dc-4e30-bdb6-a08444213d19-dbus\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:42.986252 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.986196 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:42.986346 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.986315 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jun 15 05:48:42.986590 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.986473 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:42.986735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.986712 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jun 15 05:48:42.986819 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.986757 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jun 15 05:48:42.987616 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.987599 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:42.988876 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.988859 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:42.989985 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.989964 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:42.990083 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:42.990025 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:42.991041 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.991021 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:42.991451 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.991432 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jun 15 05:48:42.992046 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.992009 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fqz8g\"" Jun 15 05:48:42.992172 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.992155 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xnm2h\"" Jun 15 05:48:42.992294 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.992276 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:42.993173 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993138 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c42wb\"" Jun 15 05:48:42.993173 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993155 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jun 15 05:48:42.993313 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993288 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jun 15 05:48:42.993361 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993311 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jun 15 05:48:42.993572 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993534 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jun 15 05:48:42.993572 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993534 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jun 15 05:48:42.993716 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993583 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jun 15 05:48:42.993716 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993658 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jun 15 05:48:42.993811 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993783 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v64lc\"" Jun 15 05:48:42.993861 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993825 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jun 15 05:48:42.993861 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.993854 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wm998\"" Jun 15 05:48:42.994059 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.994044 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jun 15 05:48:42.997570 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.997518 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jun 15 05:48:42.997666 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.997591 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l4srl\"" Jun 15 05:48:42.997666 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.997617 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jun 15 05:48:42.997779 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.997664 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jun 15 05:48:42.997779 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.997728 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jun 15 05:48:42.997961 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.997941 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jun 15 05:48:42.998054 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.997990 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jun 15 05:48:42.998112 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.998070 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pwps8\"" Jun 15 05:48:42.998185 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.998170 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jun 15 05:48:42.998369 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.998354 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jun 15 05:48:42.998405 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.998385 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jun 15 05:48:42.998745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:42.998729 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jun 15 05:48:43.005977 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.005953 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-14 05:43:41 +0000 UTC" deadline="2028-03-24 19:31:02.342240124 +0000 UTC" Jun 15 05:48:43.005977 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.005975 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15565h42m19.336267773s" Jun 15 05:48:43.021686 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.021668 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dfvrm" Jun 15 05:48:43.028994 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.028965 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dfvrm" Jun 15 05:48:43.054064 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.054012 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" event={"ID":"f281e2e25ec3badbb2247e6dadbdfaf5","Type":"ContainerStarted","Data":"8aaf740011b81fadb0c9b1f2f3175465f30d2adbf99b6413046f3a40dc49df23"} Jun 15 05:48:43.055066 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.055045 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" event={"ID":"1c5a5760696eae5c4e3ad3c580448608","Type":"ContainerStarted","Data":"3f1c0ed1a35567253a41c24598e78a62b1c2689b0c696fc93027c7140cfa29dc"} Jun 15 05:48:43.080290 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.080260 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jun 15 05:48:43.086500 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086471 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:43.086590 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvwld\" (UniqueName: \"kubernetes.io/projected/a19a3731-5372-4d37-83aa-6474aad11f0e-kube-api-access-lvwld\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.086590 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086531 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-slash\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.086590 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9j7\" (UniqueName: \"kubernetes.io/projected/4354a742-ec1d-4616-ad21-3f07511a6877-kube-api-access-gz9j7\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-etc-selinux\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98f2x\" (UniqueName: \"kubernetes.io/projected/a834d69b-c741-447e-b35c-f303b03179cf-kube-api-access-98f2x\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086633 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-cni-multus\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-systemd\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.086591 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086687 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysctl-conf\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.086724 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086708 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.086741 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret podName:9039cf21-b4dc-4e30-bdb6-a08444213d19 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:43.586720388 +0000 UTC m=+1.931923523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret") pod "global-pull-secret-syncer-pvj8k" (UID: "9039cf21-b4dc-4e30-bdb6-a08444213d19") : object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086762 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp7x4\" (UniqueName: \"kubernetes.io/projected/ad21e817-c798-46ed-a424-d9ac7ea41cb7-kube-api-access-pp7x4\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086788 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-sys\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086813 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-serviceca\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086830 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tzk2\" (UniqueName: \"kubernetes.io/projected/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-kube-api-access-4tzk2\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086845 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-socket-dir-parent\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-k8s-cni-cncf-io\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086887 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-kubelet\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-node-log\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-device-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.086952 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086934 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-host\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086959 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9039cf21-b4dc-4e30-bdb6-a08444213d19-kubelet-config\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-conf-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.086993 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19a3731-5372-4d37-83aa-6474aad11f0e-tmp-dir\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d9fg\" (UniqueName: \"kubernetes.io/projected/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-kube-api-access-8d9fg\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087028 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9039cf21-b4dc-4e30-bdb6-a08444213d19-kubelet-config\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087060 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmjgb\" (UniqueName: \"kubernetes.io/projected/691b9e0e-8935-4f07-8fc4-c3c09b653b63-kube-api-access-kmjgb\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-system-cni-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-cnibin\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087123 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-netns\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-multus-certs\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-kubelet\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-run-netns\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-ovnkube-script-lib\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087212 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-modprobe-d\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-var-lib-kubelet\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.087255 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087246 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b9d8646-aaf7-4365-8137-65a188eaecb1-cni-binary-copy\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087262 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-ovn\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087312 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-sys-fs\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087328 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysconfig\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-systemd\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087370 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-system-cni-dir\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087385 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-os-release\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087399 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/338e9902-5e59-4baf-a031-c3b32ce89c50-agent-certs\") pod \"konnectivity-agent-6z4c5\" (UID: \"338e9902-5e59-4baf-a031-c3b32ce89c50\") " pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-cni-bin\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-iptables-alerter-script\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087469 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-host-slash\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-cni-netd\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-registration-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6df76b85-5018-47b4-902b-28d79bcefb94-etc-tuned\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:43.087773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087562 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a19a3731-5372-4d37-83aa-6474aad11f0e-hosts-file\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-var-lib-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087597 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-ovnkube-config\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-env-overrides\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087669 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-daemon-config\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4354a742-ec1d-4616-ad21-3f07511a6877-ovn-node-metrics-cert\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087731 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/338e9902-5e59-4baf-a031-c3b32ce89c50-konnectivity-ca\") pod \"konnectivity-agent-6z4c5\" (UID: \"338e9902-5e59-4baf-a031-c3b32ce89c50\") " pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087749 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-socket-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysctl-d\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-lib-modules\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2wpp\" (UniqueName: \"kubernetes.io/projected/6df76b85-5018-47b4-902b-28d79bcefb94-kube-api-access-n2wpp\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087814 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-os-release\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-systemd-units\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-cni-bin\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088191 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087867 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-host\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087881 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpc6\" (UniqueName: \"kubernetes.io/projected/6b9d8646-aaf7-4365-8137-65a188eaecb1-kube-api-access-7gpc6\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087911 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-etc-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-kubernetes\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087951 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-run\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-cnibin\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.087981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-hostroot\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088028 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-log-socket\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088053 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088076 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9039cf21-b4dc-4e30-bdb6-a08444213d19-dbus\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-cni-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-etc-kubernetes\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df76b85-5018-47b4-902b-28d79bcefb94-tmp\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.088735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.088206 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9039cf21-b4dc-4e30-bdb6-a08444213d19-dbus\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:43.187733 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.187707 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jun 15 05:48:43.189290 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-cni-bin\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.189344 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189299 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-host\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.189344 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.189406 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpc6\" (UniqueName: \"kubernetes.io/projected/6b9d8646-aaf7-4365-8137-65a188eaecb1-kube-api-access-7gpc6\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.189406 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-etc-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.189406 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-kubernetes\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.189406 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189374 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-cni-bin\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.189406 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189374 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-host\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.189406 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189392 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-run\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-cnibin\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189446 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189456 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-cnibin\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189459 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-etc-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189472 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-hostroot\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189480 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-kubernetes\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189495 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-run\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-log-socket\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189518 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-hostroot\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189527 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-log-socket\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189555 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189585 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-cni-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189602 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-etc-kubernetes\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189622 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df76b85-5018-47b4-902b-28d79bcefb94-tmp\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvwld\" (UniqueName: \"kubernetes.io/projected/a19a3731-5372-4d37-83aa-6474aad11f0e-kube-api-access-lvwld\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.189679 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189680 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-etc-kubernetes\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189683 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-cni-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-slash\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9j7\" (UniqueName: \"kubernetes.io/projected/4354a742-ec1d-4616-ad21-3f07511a6877-kube-api-access-gz9j7\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189737 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-slash\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189755 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-etc-selinux\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189781 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98f2x\" (UniqueName: \"kubernetes.io/projected/a834d69b-c741-447e-b35c-f303b03179cf-kube-api-access-98f2x\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189805 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-cni-multus\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189830 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-systemd\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189873 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysctl-conf\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp7x4\" (UniqueName: \"kubernetes.io/projected/ad21e817-c798-46ed-a424-d9ac7ea41cb7-kube-api-access-pp7x4\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189938 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-cni-multus\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-sys\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-serviceca\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189978 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-systemd\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.190388 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189995 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tzk2\" (UniqueName: \"kubernetes.io/projected/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-kube-api-access-4tzk2\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-socket-dir-parent\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190038 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-k8s-cni-cncf-io\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190048 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-etc-selinux\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-kubelet\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190083 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-node-log\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190089 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190107 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-device-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190136 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190160 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-device-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190179 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.189946 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190198 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a834d69b-c741-447e-b35c-f303b03179cf-cni-binary-copy\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190231 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysctl-conf\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190236 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-sys\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190262 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-k8s-cni-cncf-io\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-host\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190274 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-kubelet\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190290 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-socket-dir-parent\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-conf-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190317 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-node-log\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190322 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-host\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190338 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19a3731-5372-4d37-83aa-6474aad11f0e-tmp-dir\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190360 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-conf-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d9fg\" (UniqueName: \"kubernetes.io/projected/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-kube-api-access-8d9fg\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190389 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190413 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmjgb\" (UniqueName: \"kubernetes.io/projected/691b9e0e-8935-4f07-8fc4-c3c09b653b63-kube-api-access-kmjgb\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190437 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-system-cni-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-cnibin\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190483 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-netns\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-multus-certs\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-kubelet\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190575 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-run-netns\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190581 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-netns\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-ovnkube-script-lib\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-serviceca\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.191920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190626 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-modprobe-d\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190632 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-cnibin\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190637 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a19a3731-5372-4d37-83aa-6474aad11f0e-tmp-dir\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190528 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-system-cni-dir\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190650 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-var-lib-kubelet\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190652 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190666 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-kubelet\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190676 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b9d8646-aaf7-4365-8137-65a188eaecb1-cni-binary-copy\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-ovn\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190714 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-var-lib-kubelet\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190719 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-run-multus-certs\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190752 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-sys-fs\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190758 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-run-ovn\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190775 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-run-netns\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-modprobe-d\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.190998 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-run-ovn-kubernetes\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191034 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysconfig\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.192799 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-systemd\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-system-cni-dir\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191113 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-sys-fs\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-os-release\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191167 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/338e9902-5e59-4baf-a031-c3b32ce89c50-agent-certs\") pod \"konnectivity-agent-6z4c5\" (UID: \"338e9902-5e59-4baf-a031-c3b32ce89c50\") " pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191178 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysconfig\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-cni-bin\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191255 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-os-release\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191298 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-systemd\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191303 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a834d69b-c741-447e-b35c-f303b03179cf-system-cni-dir\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-host-var-lib-cni-bin\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-iptables-alerter-script\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-host-slash\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191568 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6b9d8646-aaf7-4365-8137-65a188eaecb1-cni-binary-copy\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191632 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-cni-netd\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-registration-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191702 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6df76b85-5018-47b4-902b-28d79bcefb94-etc-tuned\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.193690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191740 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191773 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a19a3731-5372-4d37-83aa-6474aad11f0e-hosts-file\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191805 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-var-lib-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-ovnkube-config\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191865 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-env-overrides\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191933 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-daemon-config\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4354a742-ec1d-4616-ad21-3f07511a6877-ovn-node-metrics-cert\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/338e9902-5e59-4baf-a031-c3b32ce89c50-konnectivity-ca\") pod \"konnectivity-agent-6z4c5\" (UID: \"338e9902-5e59-4baf-a031-c3b32ce89c50\") " pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192044 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a19a3731-5372-4d37-83aa-6474aad11f0e-hosts-file\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-socket-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysctl-d\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-lib-modules\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.192154 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2wpp\" (UniqueName: \"kubernetes.io/projected/6df76b85-5018-47b4-902b-28d79bcefb94-kube-api-access-n2wpp\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.192216 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs podName:ad21e817-c798-46ed-a424-d9ac7ea41cb7 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:43.692197657 +0000 UTC m=+2.037400792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs") pod "network-metrics-daemon-pxmpj" (UID: "ad21e817-c798-46ed-a424-d9ac7ea41cb7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:43.194447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192235 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-os-release\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192304 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-systemd-units\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192406 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-systemd-units\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192463 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.192631 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-var-lib-openvswitch\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193007 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6b9d8646-aaf7-4365-8137-65a188eaecb1-multus-daemon-config\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.191931 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-cni-netd\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-registration-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193149 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4354a742-ec1d-4616-ad21-3f07511a6877-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-ovnkube-config\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193293 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/691b9e0e-8935-4f07-8fc4-c3c09b653b63-socket-dir\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193412 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-etc-sysctl-d\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193521 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6df76b85-5018-47b4-902b-28d79bcefb94-lib-modules\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193597 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-host-slash\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.193963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6df76b85-5018-47b4-902b-28d79bcefb94-tmp\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.194304 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6df76b85-5018-47b4-902b-28d79bcefb94-etc-tuned\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.194573 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/338e9902-5e59-4baf-a031-c3b32ce89c50-agent-certs\") pod \"konnectivity-agent-6z4c5\" (UID: \"338e9902-5e59-4baf-a031-c3b32ce89c50\") " pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.194686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6b9d8646-aaf7-4365-8137-65a188eaecb1-os-release\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.195288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.194733 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-env-overrides\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.196080 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.194899 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/338e9902-5e59-4baf-a031-c3b32ce89c50-konnectivity-ca\") pod \"konnectivity-agent-6z4c5\" (UID: \"338e9902-5e59-4baf-a031-c3b32ce89c50\") " pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:43.196080 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.195470 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4354a742-ec1d-4616-ad21-3f07511a6877-ovn-node-metrics-cert\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.196080 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.196027 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4354a742-ec1d-4616-ad21-3f07511a6877-ovnkube-script-lib\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.196373 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.196247 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-iptables-alerter-script\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.198687 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.198667 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 15 05:48:43.198687 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.198685 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 15 05:48:43.198862 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.198695 2569 projected.go:196] Error preparing data for projected volume kube-api-access-h2tbj for pod openshift-network-diagnostics/network-check-target-8n9hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:43.198862 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.198750 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj podName:e8db85b7-db76-4d90-830a-af7729764788 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:43.698737582 +0000 UTC m=+2.043940712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h2tbj" (UniqueName: "kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj") pod "network-check-target-8n9hm" (UID: "e8db85b7-db76-4d90-830a-af7729764788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:43.200392 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.200360 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvwld\" (UniqueName: \"kubernetes.io/projected/a19a3731-5372-4d37-83aa-6474aad11f0e-kube-api-access-lvwld\") pod \"node-resolver-zqzpb\" (UID: \"a19a3731-5372-4d37-83aa-6474aad11f0e\") " pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.201795 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.201653 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmjgb\" (UniqueName: \"kubernetes.io/projected/691b9e0e-8935-4f07-8fc4-c3c09b653b63-kube-api-access-kmjgb\") pod \"aws-ebs-csi-driver-node-hf54t\" (UID: \"691b9e0e-8935-4f07-8fc4-c3c09b653b63\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.201795 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.201749 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d9fg\" (UniqueName: \"kubernetes.io/projected/db6f95d9-efb8-4a79-9abd-ce9d4d485d95-kube-api-access-8d9fg\") pod \"iptables-alerter-747dl\" (UID: \"db6f95d9-efb8-4a79-9abd-ce9d4d485d95\") " pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.202177 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.202147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp7x4\" (UniqueName: \"kubernetes.io/projected/ad21e817-c798-46ed-a424-d9ac7ea41cb7-kube-api-access-pp7x4\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:43.202400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.202375 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2wpp\" (UniqueName: \"kubernetes.io/projected/6df76b85-5018-47b4-902b-28d79bcefb94-kube-api-access-n2wpp\") pod \"tuned-zzm2s\" (UID: \"6df76b85-5018-47b4-902b-28d79bcefb94\") " pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.202596 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.202575 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98f2x\" (UniqueName: \"kubernetes.io/projected/a834d69b-c741-447e-b35c-f303b03179cf-kube-api-access-98f2x\") pod \"multus-additional-cni-plugins-v2kln\" (UID: \"a834d69b-c741-447e-b35c-f303b03179cf\") " pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.202661 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.202644 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpc6\" (UniqueName: \"kubernetes.io/projected/6b9d8646-aaf7-4365-8137-65a188eaecb1-kube-api-access-7gpc6\") pod \"multus-nt2fx\" (UID: \"6b9d8646-aaf7-4365-8137-65a188eaecb1\") " pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.203112 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.203097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9j7\" (UniqueName: \"kubernetes.io/projected/4354a742-ec1d-4616-ad21-3f07511a6877-kube-api-access-gz9j7\") pod \"ovnkube-node-rkdkp\" (UID: \"4354a742-ec1d-4616-ad21-3f07511a6877\") " pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.203184 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.203169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tzk2\" (UniqueName: \"kubernetes.io/projected/2be932e7-3cd4-48b1-88ac-e9cead0e0a1e-kube-api-access-4tzk2\") pod \"node-ca-g28pn\" (UID: \"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e\") " pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.291582 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.291457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:48:43.298743 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.298716 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" Jun 15 05:48:43.304695 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.304576 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" Jun 15 05:48:43.304922 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:43.304897 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338e9902_5e59_4baf_a031_c3b32ce89c50.slice/crio-565898a9bcb254842b6a3b339289b2de2228ff814cd56f812b242370b54eb8c7 WatchSource:0}: Error finding container 565898a9bcb254842b6a3b339289b2de2228ff814cd56f812b242370b54eb8c7: Status 404 returned error can't find the container with id 565898a9bcb254842b6a3b339289b2de2228ff814cd56f812b242370b54eb8c7 Jun 15 05:48:43.308563 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:43.308499 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691b9e0e_8935_4f07_8fc4_c3c09b653b63.slice/crio-92f40227c06c729d722f14de2f19bd0bdf02ed9bb437cd6d108e72d24372fd11 WatchSource:0}: Error finding container 92f40227c06c729d722f14de2f19bd0bdf02ed9bb437cd6d108e72d24372fd11: Status 404 returned error can't find the container with id 92f40227c06c729d722f14de2f19bd0bdf02ed9bb437cd6d108e72d24372fd11 Jun 15 05:48:43.309596 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.309573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nt2fx" Jun 15 05:48:43.318280 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.318146 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqzpb" Jun 15 05:48:43.322703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.322680 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g28pn" Jun 15 05:48:43.327594 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.327563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v2kln" Jun 15 05:48:43.331695 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:43.331669 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19a3731_5372_4d37_83aa_6474aad11f0e.slice/crio-0572983a749ec98847ef2190d51f4404d1c196774a61f97958e29d8a7cb858ad WatchSource:0}: Error finding container 0572983a749ec98847ef2190d51f4404d1c196774a61f97958e29d8a7cb858ad: Status 404 returned error can't find the container with id 0572983a749ec98847ef2190d51f4404d1c196774a61f97958e29d8a7cb858ad Jun 15 05:48:43.334664 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.334252 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-747dl" Jun 15 05:48:43.339534 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.339350 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:48:43.339883 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:43.339832 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be932e7_3cd4_48b1_88ac_e9cead0e0a1e.slice/crio-7f8eeb4202375ad4406e545c5dc81a3db5e851878dc7a35efc09752ab89bacf4 WatchSource:0}: Error finding container 7f8eeb4202375ad4406e545c5dc81a3db5e851878dc7a35efc09752ab89bacf4: Status 404 returned error can't find the container with id 7f8eeb4202375ad4406e545c5dc81a3db5e851878dc7a35efc09752ab89bacf4 Jun 15 05:48:43.344278 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:43.344245 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda834d69b_c741_447e_b35c_f303b03179cf.slice/crio-11436404a20ec6508de2dcb0cfc924d76df29a6ad4655c25782733449149077f WatchSource:0}: Error finding container 11436404a20ec6508de2dcb0cfc924d76df29a6ad4655c25782733449149077f: Status 404 returned error can't find the container with id 11436404a20ec6508de2dcb0cfc924d76df29a6ad4655c25782733449149077f Jun 15 05:48:43.349435 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:43.349412 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6f95d9_efb8_4a79_9abd_ce9d4d485d95.slice/crio-99b175866b78838ca90f6064f9051166a318f6872d37bb68efad1e36a9aff892 WatchSource:0}: Error finding container 99b175866b78838ca90f6064f9051166a318f6872d37bb68efad1e36a9aff892: Status 404 returned error can't find the container with id 99b175866b78838ca90f6064f9051166a318f6872d37bb68efad1e36a9aff892 Jun 15 05:48:43.358335 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:48:43.358310 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4354a742_ec1d_4616_ad21_3f07511a6877.slice/crio-3b4b9c21bc1a7952398817a55f44b37e442d416a28975c450ac8a3cb494e3a4c WatchSource:0}: Error finding container 3b4b9c21bc1a7952398817a55f44b37e442d416a28975c450ac8a3cb494e3a4c: Status 404 returned error can't find the container with id 3b4b9c21bc1a7952398817a55f44b37e442d416a28975c450ac8a3cb494e3a4c Jun 15 05:48:43.595928 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.595271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:43.595928 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.595439 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:43.595928 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.595502 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret podName:9039cf21-b4dc-4e30-bdb6-a08444213d19 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:44.595483855 +0000 UTC m=+2.940686974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret") pod "global-pull-secret-syncer-pvj8k" (UID: "9039cf21-b4dc-4e30-bdb6-a08444213d19") : object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:43.697360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.696657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:43.697360 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.696957 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:43.697360 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.697028 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs podName:ad21e817-c798-46ed-a424-d9ac7ea41cb7 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:44.697008724 +0000 UTC m=+3.042211855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs") pod "network-metrics-daemon-pxmpj" (UID: "ad21e817-c798-46ed-a424-d9ac7ea41cb7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:43.797769 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:43.797146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:43.797769 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.797319 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 15 05:48:43.797769 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.797339 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 15 05:48:43.797769 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.797352 2569 projected.go:196] Error preparing data for projected volume kube-api-access-h2tbj for pod openshift-network-diagnostics/network-check-target-8n9hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:43.797769 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:43.797414 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj podName:e8db85b7-db76-4d90-830a-af7729764788 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:44.797395741 +0000 UTC m=+3.142598881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2tbj" (UniqueName: "kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj") pod "network-check-target-8n9hm" (UID: "e8db85b7-db76-4d90-830a-af7729764788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:44.031904 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.031870 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:43:43 +0000 UTC" deadline="2027-12-15 05:38:13.001195118 +0000 UTC" Jun 15 05:48:44.032448 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.031909 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13151h49m28.969290423s" Jun 15 05:48:44.066337 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.066297 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"3b4b9c21bc1a7952398817a55f44b37e442d416a28975c450ac8a3cb494e3a4c"} Jun 15 05:48:44.079089 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.079016 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g28pn" event={"ID":"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e","Type":"ContainerStarted","Data":"7f8eeb4202375ad4406e545c5dc81a3db5e851878dc7a35efc09752ab89bacf4"} Jun 15 05:48:44.083052 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.083019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqzpb" event={"ID":"a19a3731-5372-4d37-83aa-6474aad11f0e","Type":"ContainerStarted","Data":"0572983a749ec98847ef2190d51f4404d1c196774a61f97958e29d8a7cb858ad"} Jun 15 05:48:44.097404 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.097370 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" event={"ID":"6df76b85-5018-47b4-902b-28d79bcefb94","Type":"ContainerStarted","Data":"f53cb373d4e45d76d115742500696ccf372eb4fb7be3e55554f8c39034aaaa62"} Jun 15 05:48:44.111996 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.111868 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" event={"ID":"691b9e0e-8935-4f07-8fc4-c3c09b653b63","Type":"ContainerStarted","Data":"92f40227c06c729d722f14de2f19bd0bdf02ed9bb437cd6d108e72d24372fd11"} Jun 15 05:48:44.117037 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.117004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6z4c5" event={"ID":"338e9902-5e59-4baf-a031-c3b32ce89c50","Type":"ContainerStarted","Data":"565898a9bcb254842b6a3b339289b2de2228ff814cd56f812b242370b54eb8c7"} Jun 15 05:48:44.126563 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.126321 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-747dl" event={"ID":"db6f95d9-efb8-4a79-9abd-ce9d4d485d95","Type":"ContainerStarted","Data":"99b175866b78838ca90f6064f9051166a318f6872d37bb68efad1e36a9aff892"} Jun 15 05:48:44.135265 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.135228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerStarted","Data":"11436404a20ec6508de2dcb0cfc924d76df29a6ad4655c25782733449149077f"} Jun 15 05:48:44.144482 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.144447 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nt2fx" event={"ID":"6b9d8646-aaf7-4365-8137-65a188eaecb1","Type":"ContainerStarted","Data":"5120eafcaa4eb6e2a1a9984c35b859baf97a797c181afa1414e87c08df96a43d"} Jun 15 05:48:44.266268 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.266236 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jun 15 05:48:44.606737 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.606701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:44.606938 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.606878 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:44.606994 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.606975 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret podName:9039cf21-b4dc-4e30-bdb6-a08444213d19 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:46.606954037 +0000 UTC m=+4.952157211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret") pod "global-pull-secret-syncer-pvj8k" (UID: "9039cf21-b4dc-4e30-bdb6-a08444213d19") : object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:44.708052 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.708014 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:44.708267 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.708166 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:44.708267 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.708244 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs podName:ad21e817-c798-46ed-a424-d9ac7ea41cb7 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:46.708225873 +0000 UTC m=+5.053429007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs") pod "network-metrics-daemon-pxmpj" (UID: "ad21e817-c798-46ed-a424-d9ac7ea41cb7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:44.808436 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:44.808397 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:44.808685 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.808630 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 15 05:48:44.808685 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.808653 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 15 05:48:44.808685 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.808666 2569 projected.go:196] Error preparing data for projected volume kube-api-access-h2tbj for pod openshift-network-diagnostics/network-check-target-8n9hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:44.808864 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:44.808724 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj podName:e8db85b7-db76-4d90-830a-af7729764788 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:46.808705996 +0000 UTC m=+5.153909136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2tbj" (UniqueName: "kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj") pod "network-check-target-8n9hm" (UID: "e8db85b7-db76-4d90-830a-af7729764788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:45.032671 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:45.032631 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:43:43 +0000 UTC" deadline="2028-03-22 06:06:05.893430486 +0000 UTC" Jun 15 05:48:45.032671 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:45.032673 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15504h17m20.860762248s" Jun 15 05:48:45.051387 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:45.051355 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:45.051652 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:45.051491 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:45.051652 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:45.051598 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:45.051792 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:45.051669 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:45.051792 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:45.051724 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:45.051896 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:45.051789 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:45.703626 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:45.703290 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jun 15 05:48:46.622634 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:46.622473 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:46.622634 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.622643 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:46.623237 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.622712 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret podName:9039cf21-b4dc-4e30-bdb6-a08444213d19 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:50.62269411 +0000 UTC m=+8.967897230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret") pod "global-pull-secret-syncer-pvj8k" (UID: "9039cf21-b4dc-4e30-bdb6-a08444213d19") : object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:46.722876 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:46.722841 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:46.723063 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.722963 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:46.723063 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.723020 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs podName:ad21e817-c798-46ed-a424-d9ac7ea41cb7 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:50.723000739 +0000 UTC m=+9.068203877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs") pod "network-metrics-daemon-pxmpj" (UID: "ad21e817-c798-46ed-a424-d9ac7ea41cb7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:46.824840 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:46.824220 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:46.824840 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.824392 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 15 05:48:46.824840 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.824412 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 15 05:48:46.824840 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.824424 2569 projected.go:196] Error preparing data for projected volume kube-api-access-h2tbj for pod openshift-network-diagnostics/network-check-target-8n9hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:46.824840 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:46.824483 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj podName:e8db85b7-db76-4d90-830a-af7729764788 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:50.824463601 +0000 UTC m=+9.169666723 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2tbj" (UniqueName: "kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj") pod "network-check-target-8n9hm" (UID: "e8db85b7-db76-4d90-830a-af7729764788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:47.051122 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:47.050643 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:47.051122 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:47.050686 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:47.051122 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:47.050646 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:47.051122 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:47.050776 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:47.051122 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:47.050888 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:47.051122 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:47.050961 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:49.051438 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:49.051178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:49.051924 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:49.051242 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:49.051924 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:49.051525 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:49.051924 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:49.051242 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:49.051924 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:49.051631 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:49.051924 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:49.051743 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:50.654921 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:50.654866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:50.655445 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.655125 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:50.655445 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.655210 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret podName:9039cf21-b4dc-4e30-bdb6-a08444213d19 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:58.655178073 +0000 UTC m=+17.000381192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret") pod "global-pull-secret-syncer-pvj8k" (UID: "9039cf21-b4dc-4e30-bdb6-a08444213d19") : object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:50.756363 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:50.756314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:50.756575 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.756491 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:50.756642 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.756625 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs podName:ad21e817-c798-46ed-a424-d9ac7ea41cb7 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:58.756602125 +0000 UTC m=+17.101805246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs") pod "network-metrics-daemon-pxmpj" (UID: "ad21e817-c798-46ed-a424-d9ac7ea41cb7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:50.857844 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:50.857222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:50.857844 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.857400 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 15 05:48:50.857844 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.857419 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 15 05:48:50.857844 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.857433 2569 projected.go:196] Error preparing data for projected volume kube-api-access-h2tbj for pod openshift-network-diagnostics/network-check-target-8n9hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:50.857844 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:50.857501 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj podName:e8db85b7-db76-4d90-830a-af7729764788 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:58.857482128 +0000 UTC m=+17.202685250 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2tbj" (UniqueName: "kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj") pod "network-check-target-8n9hm" (UID: "e8db85b7-db76-4d90-830a-af7729764788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:51.051069 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:51.051023 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:51.051256 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:51.051170 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:51.051256 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:51.051204 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:51.051573 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:51.051371 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:51.051573 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:51.051398 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:51.051573 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:51.051514 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:53.050925 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:53.050827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:53.050925 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:53.050876 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:53.051451 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:53.050967 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:53.051451 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:53.051140 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:53.051451 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:53.051248 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:53.051451 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:53.051329 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:55.051206 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:55.051164 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:55.051665 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:55.051164 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:55.051665 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:55.051279 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:55.051665 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:55.051378 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:55.051665 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:55.051171 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:55.051665 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:55.051489 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:57.050739 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:57.050702 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:57.051182 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:57.050835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:57.051182 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:57.050848 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:57.051182 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:57.050846 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:57.051182 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:57.050952 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:48:57.051182 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:57.051046 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:58.710900 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:58.710866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:58.711279 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.711010 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:58.711279 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.711069 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret podName:9039cf21-b4dc-4e30-bdb6-a08444213d19 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:14.711055528 +0000 UTC m=+33.056258650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret") pod "global-pull-secret-syncer-pvj8k" (UID: "9039cf21-b4dc-4e30-bdb6-a08444213d19") : object "kube-system"/"original-pull-secret" not registered Jun 15 05:48:58.811212 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:58.811179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:58.811344 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.811322 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:58.811397 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.811388 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs podName:ad21e817-c798-46ed-a424-d9ac7ea41cb7 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:14.811373608 +0000 UTC m=+33.156576727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs") pod "network-metrics-daemon-pxmpj" (UID: "ad21e817-c798-46ed-a424-d9ac7ea41cb7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:48:58.911869 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:58.911841 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:58.912014 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.911994 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 15 05:48:58.912014 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.912015 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 15 05:48:58.912084 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.912026 2569 projected.go:196] Error preparing data for projected volume kube-api-access-h2tbj for pod openshift-network-diagnostics/network-check-target-8n9hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:58.912084 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:58.912074 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj podName:e8db85b7-db76-4d90-830a-af7729764788 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:14.912061223 +0000 UTC m=+33.257264343 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2tbj" (UniqueName: "kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj") pod "network-check-target-8n9hm" (UID: "e8db85b7-db76-4d90-830a-af7729764788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:48:59.050585 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:59.050536 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:48:59.050585 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:59.050576 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:48:59.050823 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:48:59.050553 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:48:59.050823 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:59.050679 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:48:59.050823 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:59.050736 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:48:59.050823 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:48:59.050809 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:01.050995 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:01.050958 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:01.050995 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:01.050980 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:01.050995 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:01.050993 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:01.051497 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:01.051100 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:01.051497 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:01.051246 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:01.051497 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:01.051322 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:03.051305 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:03.051272 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:03.051829 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:03.051372 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:03.051829 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:03.051387 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:03.051829 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:03.051460 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:03.051829 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:03.051493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:03.051829 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:03.051579 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:04.195298 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.195076 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nt2fx" event={"ID":"6b9d8646-aaf7-4365-8137-65a188eaecb1","Type":"ContainerStarted","Data":"38a5893eadccf2ddc4438c199db94f504317282a2ab4533f55c694d01d93ce93"} Jun 15 05:49:04.196531 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.196511 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" event={"ID":"f281e2e25ec3badbb2247e6dadbdfaf5","Type":"ContainerStarted","Data":"dcae4d0013cc62f8f91b3fb463b30d1d4faa32ec96580276ed00a376652ef28f"} Jun 15 05:49:04.199199 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199182 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:49:04.199492 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199470 2569 generic.go:356] "Generic (PLEG): container finished" podID="4354a742-ec1d-4616-ad21-3f07511a6877" containerID="3b8a24b062dda43e9f0344d4a458a03371da2b9f36e764e29588d7ca39ac305d" exitCode=1 Jun 15 05:49:04.199589 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"92689dd9bd83e9e20f39602193d8024faefd8ee2eaf7be31aa04a9b0531a7cbf"} Jun 15 05:49:04.199659 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199618 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"5152610f6f24d80afc614fc72278c20ba0689d30b37e510077fcbc7e9f112054"} Jun 15 05:49:04.199718 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199689 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"dc96ea7fff1c53fcfca0c7a1017a555b03c1aeba4373800e7435d4113750248d"} Jun 15 05:49:04.199772 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199738 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"298f9fa9533d411c3c591431eb4ee253622ba3748ad8d1eb815bf37e7df6f225"} Jun 15 05:49:04.199826 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199785 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerDied","Data":"3b8a24b062dda43e9f0344d4a458a03371da2b9f36e764e29588d7ca39ac305d"} Jun 15 05:49:04.199885 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.199836 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"2a8189fd4472ecb62d58842aa21517b641797b10eb6269a8756d619313425044"} Jun 15 05:49:04.200955 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.200936 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" event={"ID":"6df76b85-5018-47b4-902b-28d79bcefb94","Type":"ContainerStarted","Data":"664c96a0501c5750a0ffd698bef2892a2a9a293e92c8c8f3d4a7dc8f17441e3d"} Jun 15 05:49:04.218355 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.218310 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nt2fx" podStartSLOduration=1.8462498950000001 podStartE2EDuration="22.218296545s" totalImagesPullingTime="20.37204665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:04.218190323 +0000 UTC m=+22.563393463" watchObservedRunningTime="2026-06-15 05:49:04.218296545 +0000 UTC m=+22.563499686" Jun 15 05:49:04.237673 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.237598 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zzm2s" podStartSLOduration=2.362244027 podStartE2EDuration="22.237576861s" totalImagesPullingTime="19.875332834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:04.237142925 +0000 UTC m=+22.582346083" watchObservedRunningTime="2026-06-15 05:49:04.237576861 +0000 UTC m=+22.582780002" Jun 15 05:49:04.255099 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:04.254917 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-226.ec2.internal" podStartSLOduration=22.254899272 podStartE2EDuration="22.254899272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:04.254663331 +0000 UTC m=+22.599866485" watchObservedRunningTime="2026-06-15 05:49:04.254899272 +0000 UTC m=+22.600102412" Jun 15 05:49:05.051160 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.051128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:05.051320 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.051222 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:05.051320 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.051248 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:05.051408 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:05.051319 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:05.051408 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:05.051226 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:05.051408 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:05.051394 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:05.204051 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.204017 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g28pn" event={"ID":"2be932e7-3cd4-48b1-88ac-e9cead0e0a1e","Type":"ContainerStarted","Data":"750ce4113452364b6d9fa5fdffb73305bd1535c7ac894f9c8b01496fa867e713"} Jun 15 05:49:05.205471 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.205443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqzpb" event={"ID":"a19a3731-5372-4d37-83aa-6474aad11f0e","Type":"ContainerStarted","Data":"45a41a129881c83b687590ee403cd3c78d96e7bacca4770a7d53ab1b203fe560"} Jun 15 05:49:05.206808 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.206788 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" event={"ID":"691b9e0e-8935-4f07-8fc4-c3c09b653b63","Type":"ContainerStarted","Data":"b281fbbf6ba3395f3dafa169af01cf0fbebba3f7162fd837f91dc696126b39b4"} Jun 15 05:49:05.208042 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.208020 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6z4c5" event={"ID":"338e9902-5e59-4baf-a031-c3b32ce89c50","Type":"ContainerStarted","Data":"a93c1c211a1502625a314e39192561202fae22b6066364164f8c42fc745d76ed"} Jun 15 05:49:05.209458 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.209428 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-747dl" event={"ID":"db6f95d9-efb8-4a79-9abd-ce9d4d485d95","Type":"ContainerStarted","Data":"b5a5e3e1b1e1b577f12e2f7518aa39f81d0ec01d3e071c98d176fe55346447d5"} Jun 15 05:49:05.210834 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.210814 2569 generic.go:356] "Generic (PLEG): container finished" podID="a834d69b-c741-447e-b35c-f303b03179cf" containerID="cbafbe3bb5ac44b7e6d253f5197ed353aa40d4cb00854e4834af89ab1366e964" exitCode=0 Jun 15 05:49:05.210924 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.210877 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerDied","Data":"cbafbe3bb5ac44b7e6d253f5197ed353aa40d4cb00854e4834af89ab1366e964"} Jun 15 05:49:05.212276 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.212257 2569 generic.go:356] "Generic (PLEG): container finished" podID="1c5a5760696eae5c4e3ad3c580448608" containerID="0f80b730fee5c9fa5c4c2caa4b58987f2d90efb86281eb3fb2671d7e66debfb6" exitCode=0 Jun 15 05:49:05.212376 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.212350 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" event={"ID":"1c5a5760696eae5c4e3ad3c580448608","Type":"ContainerDied","Data":"0f80b730fee5c9fa5c4c2caa4b58987f2d90efb86281eb3fb2671d7e66debfb6"} Jun 15 05:49:05.238262 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.238220 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g28pn" podStartSLOduration=3.418769274 podStartE2EDuration="23.238207002s" totalImagesPullingTime="19.819437728s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:05.22261178 +0000 UTC m=+23.567814922" watchObservedRunningTime="2026-06-15 05:49:05.238207002 +0000 UTC m=+23.583410143" Jun 15 05:49:05.256179 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.256140 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6z4c5" podStartSLOduration=3.40088166 podStartE2EDuration="23.256127067s" totalImagesPullingTime="19.855245407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:05.25578974 +0000 UTC m=+23.600992881" watchObservedRunningTime="2026-06-15 05:49:05.256127067 +0000 UTC m=+23.601330205" Jun 15 05:49:05.276105 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.276066 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zqzpb" podStartSLOduration=3.411767081 podStartE2EDuration="23.276049983s" totalImagesPullingTime="19.864282902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:05.275745206 +0000 UTC m=+23.620948347" watchObservedRunningTime="2026-06-15 05:49:05.276049983 +0000 UTC m=+23.621253126" Jun 15 05:49:05.315288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.315192 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-747dl" podStartSLOduration=3.505687021 podStartE2EDuration="23.315177082s" totalImagesPullingTime="19.809490061s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:05.314614032 +0000 UTC m=+23.659817172" watchObservedRunningTime="2026-06-15 05:49:05.315177082 +0000 UTC m=+23.660380222" Jun 15 05:49:05.539050 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:05.538965 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jun 15 05:49:06.052449 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:06.052325 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-15T05:49:05.538982489Z","UUID":"3b7e15f2-bafe-420d-b6b1-c59211f3f3aa","Handler":null,"Name":"","Endpoint":""} Jun 15 05:49:06.054699 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:06.054678 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jun 15 05:49:06.054842 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:06.054723 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jun 15 05:49:06.218030 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:06.217993 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" event={"ID":"691b9e0e-8935-4f07-8fc4-c3c09b653b63","Type":"ContainerStarted","Data":"e4558ccb30e7bc698516262d84712eb65bda2927021a3cb22113666e546570a7"} Jun 15 05:49:06.220347 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:06.220290 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" event={"ID":"1c5a5760696eae5c4e3ad3c580448608","Type":"ContainerStarted","Data":"39c597326f973dc810124b3596ff82b68cedf10706d460b31213f182c81825b2"} Jun 15 05:49:07.051201 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.051168 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:07.051201 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.051204 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:07.051525 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.051293 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:07.051525 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:07.051290 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:07.051525 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:07.051385 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:07.051525 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:07.051458 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:07.226457 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.226426 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:49:07.227025 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.226907 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"9d01ccaf9fe6996f65b4432f6c37fd455240dcb326b55f9ee02ebfc0d5289225"} Jun 15 05:49:07.229163 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.229138 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" event={"ID":"691b9e0e-8935-4f07-8fc4-c3c09b653b63","Type":"ContainerStarted","Data":"86d8c45093795d7ab470e361f771da819a20f681ddd932f8ecaed647fa59f7ff"} Jun 15 05:49:07.250030 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.249975 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hf54t" podStartSLOduration=3.395961961 podStartE2EDuration="25.249957252s" totalImagesPullingTime="21.853995291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:07.249493922 +0000 UTC m=+25.594697077" watchObservedRunningTime="2026-06-15 05:49:07.249957252 +0000 UTC m=+25.595160394" Jun 15 05:49:07.250223 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:07.250198 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-226.ec2.internal" podStartSLOduration=25.250191708 podStartE2EDuration="25.250191708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:06.238602611 +0000 UTC m=+24.583805755" watchObservedRunningTime="2026-06-15 05:49:07.250191708 +0000 UTC m=+25.595394850" Jun 15 05:49:09.051575 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:09.051329 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:09.052057 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:09.051388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:09.052057 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:09.051672 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:09.052057 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:09.051388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:09.052057 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:09.051778 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:09.052057 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:09.051866 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:09.238536 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:09.238512 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:49:10.043040 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.043007 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:49:10.043584 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.043554 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:49:10.242216 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.242180 2569 generic.go:356] "Generic (PLEG): container finished" podID="a834d69b-c741-447e-b35c-f303b03179cf" containerID="6457553fe960095ed50aeeb094f4ef46803f630f23bbf70da528fbcd10e79735" exitCode=0 Jun 15 05:49:10.242899 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.242264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerDied","Data":"6457553fe960095ed50aeeb094f4ef46803f630f23bbf70da528fbcd10e79735"} Jun 15 05:49:10.245932 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.245775 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:49:10.246334 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.246306 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"3e65af43e926cc2ac0896af3158459b02d55530bda72804487f92ac91d2ee253"} Jun 15 05:49:10.246580 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.246567 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:49:10.246819 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.246803 2569 scope.go:117] "RemoveContainer" containerID="3b8a24b062dda43e9f0344d4a458a03371da2b9f36e764e29588d7ca39ac305d" Jun 15 05:49:10.246969 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:10.246952 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6z4c5" Jun 15 05:49:11.050811 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.050780 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:11.051185 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.050792 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:11.051185 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.050891 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:11.051185 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:11.050877 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:11.051185 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:11.050969 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:11.051185 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:11.051042 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:11.254891 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.254859 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:49:11.255615 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.255584 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" event={"ID":"4354a742-ec1d-4616-ad21-3f07511a6877","Type":"ContainerStarted","Data":"5d2804784a67a09615256eb1fafd9a445ed3acd6620411ba947f97ca73e9aadf"} Jun 15 05:49:11.255833 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.255812 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:49:11.255900 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.255888 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:49:11.256047 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.256032 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:49:11.277352 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.277322 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:49:11.278873 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.278848 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:49:11.295054 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.295002 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" podStartSLOduration=9.230855261 podStartE2EDuration="29.294986312s" totalImagesPullingTime="20.064131051s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:11.294450408 +0000 UTC m=+29.639653574" watchObservedRunningTime="2026-06-15 05:49:11.294986312 +0000 UTC m=+29.640189453" Jun 15 05:49:11.514499 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.514468 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8n9hm"] Jun 15 05:49:11.514655 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.514595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:11.514699 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:11.514672 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:11.518069 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.518038 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvj8k"] Jun 15 05:49:11.518221 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.518159 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:11.518286 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:11.518255 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:11.518728 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.518703 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pxmpj"] Jun 15 05:49:11.518862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:11.518829 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:11.518955 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:11.518935 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:12.259901 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:12.259865 2569 generic.go:356] "Generic (PLEG): container finished" podID="a834d69b-c741-447e-b35c-f303b03179cf" containerID="360513105c16728b024c3f3ea20107fca0cd6a80f271d3fcbab9544e44b61193" exitCode=0 Jun 15 05:49:12.260325 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:12.259952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerDied","Data":"360513105c16728b024c3f3ea20107fca0cd6a80f271d3fcbab9544e44b61193"} Jun 15 05:49:13.051087 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:13.051055 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:13.051087 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:13.051078 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:13.051247 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:13.051073 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:13.051247 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:13.051176 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:13.051247 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:13.051227 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:13.051360 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:13.051276 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:13.264885 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:13.264804 2569 generic.go:356] "Generic (PLEG): container finished" podID="a834d69b-c741-447e-b35c-f303b03179cf" containerID="1f95f79a44b998e35a69d048c0d9a9ecf7c08a47cb0ced0c7244594a4f696455" exitCode=0 Jun 15 05:49:13.265232 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:13.264893 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerDied","Data":"1f95f79a44b998e35a69d048c0d9a9ecf7c08a47cb0ced0c7244594a4f696455"} Jun 15 05:49:14.721878 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:14.721844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:14.722300 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.722010 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jun 15 05:49:14.722300 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.722116 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret podName:9039cf21-b4dc-4e30-bdb6-a08444213d19 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:46.722075675 +0000 UTC m=+65.067278797 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret") pod "global-pull-secret-syncer-pvj8k" (UID: "9039cf21-b4dc-4e30-bdb6-a08444213d19") : object "kube-system"/"original-pull-secret" not registered Jun 15 05:49:14.823046 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:14.823004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:14.823235 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.823172 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:49:14.823314 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.823246 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs podName:ad21e817-c798-46ed-a424-d9ac7ea41cb7 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:46.823230196 +0000 UTC m=+65.168433316 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs") pod "network-metrics-daemon-pxmpj" (UID: "ad21e817-c798-46ed-a424-d9ac7ea41cb7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jun 15 05:49:14.923998 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:14.923961 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:14.924166 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.924100 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jun 15 05:49:14.924166 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.924118 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jun 15 05:49:14.924166 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.924129 2569 projected.go:196] Error preparing data for projected volume kube-api-access-h2tbj for pod openshift-network-diagnostics/network-check-target-8n9hm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:49:14.924297 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:14.924190 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj podName:e8db85b7-db76-4d90-830a-af7729764788 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:46.924171941 +0000 UTC m=+65.269375083 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h2tbj" (UniqueName: "kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj") pod "network-check-target-8n9hm" (UID: "e8db85b7-db76-4d90-830a-af7729764788") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jun 15 05:49:15.050918 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:15.050885 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:15.050918 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:15.050911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:15.051351 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:15.050911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:15.051351 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:15.051043 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:15.051351 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:15.051189 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:15.051351 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:15.051304 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:17.051446 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.051406 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:17.051446 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.051445 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:17.051965 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.051445 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:17.051965 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:17.051555 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvj8k" podUID="9039cf21-b4dc-4e30-bdb6-a08444213d19" Jun 15 05:49:17.051965 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:17.051663 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pxmpj" podUID="ad21e817-c798-46ed-a424-d9ac7ea41cb7" Jun 15 05:49:17.051965 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:17.051738 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8n9hm" podUID="e8db85b7-db76-4d90-830a-af7729764788" Jun 15 05:49:17.498003 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.497973 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-226.ec2.internal" event="NodeReady" Jun 15 05:49:17.498129 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.498099 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jun 15 05:49:17.549198 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.549165 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fmtjs"] Jun 15 05:49:17.569930 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.569900 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d97j4"] Jun 15 05:49:17.570107 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.569964 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.572960 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.572910 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jun 15 05:49:17.573259 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.573236 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-f5wbk\"" Jun 15 05:49:17.573372 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.573242 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jun 15 05:49:17.573372 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.573282 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jun 15 05:49:17.593954 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.593922 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmtjs"] Jun 15 05:49:17.593954 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.593959 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d97j4"] Jun 15 05:49:17.594176 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.594068 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.596374 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.596350 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jun 15 05:49:17.596495 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.596379 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bq244\"" Jun 15 05:49:17.596495 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.596427 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jun 15 05:49:17.645212 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.645153 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312ffed8-5109-4ae0-9b62-74f951fc8068-cert\") pod \"ingress-canary-fmtjs\" (UID: \"312ffed8-5109-4ae0-9b62-74f951fc8068\") " pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.645321 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.645211 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6787\" (UniqueName: \"kubernetes.io/projected/312ffed8-5109-4ae0-9b62-74f951fc8068-kube-api-access-m6787\") pod \"ingress-canary-fmtjs\" (UID: \"312ffed8-5109-4ae0-9b62-74f951fc8068\") " pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.745935 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.745899 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c07138-376d-46c4-9ac5-85733a50439a-tmp-dir\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.746074 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.745964 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c07138-376d-46c4-9ac5-85733a50439a-config-volume\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.746074 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.745981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf6x8\" (UniqueName: \"kubernetes.io/projected/d4c07138-376d-46c4-9ac5-85733a50439a-kube-api-access-pf6x8\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.746074 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.746017 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312ffed8-5109-4ae0-9b62-74f951fc8068-cert\") pod \"ingress-canary-fmtjs\" (UID: \"312ffed8-5109-4ae0-9b62-74f951fc8068\") " pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.746074 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.746061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.746215 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.746083 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6787\" (UniqueName: \"kubernetes.io/projected/312ffed8-5109-4ae0-9b62-74f951fc8068-kube-api-access-m6787\") pod \"ingress-canary-fmtjs\" (UID: \"312ffed8-5109-4ae0-9b62-74f951fc8068\") " pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.749995 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.749974 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/312ffed8-5109-4ae0-9b62-74f951fc8068-cert\") pod \"ingress-canary-fmtjs\" (UID: \"312ffed8-5109-4ae0-9b62-74f951fc8068\") " pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.760477 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.760452 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6787\" (UniqueName: \"kubernetes.io/projected/312ffed8-5109-4ae0-9b62-74f951fc8068-kube-api-access-m6787\") pod \"ingress-canary-fmtjs\" (UID: \"312ffed8-5109-4ae0-9b62-74f951fc8068\") " pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.847098 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.847064 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c07138-376d-46c4-9ac5-85733a50439a-config-volume\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.847098 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.847102 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf6x8\" (UniqueName: \"kubernetes.io/projected/d4c07138-376d-46c4-9ac5-85733a50439a-kube-api-access-pf6x8\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.847323 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.847163 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.847323 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.847221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c07138-376d-46c4-9ac5-85733a50439a-tmp-dir\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.847323 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:17.847294 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 15 05:49:17.847471 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:17.847357 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls podName:d4c07138-376d-46c4-9ac5-85733a50439a nodeName:}" failed. No retries permitted until 2026-06-15 05:49:18.347340505 +0000 UTC m=+36.692543629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls") pod "dns-default-d97j4" (UID: "d4c07138-376d-46c4-9ac5-85733a50439a") : secret "dns-default-metrics-tls" not found Jun 15 05:49:17.847531 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.847511 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4c07138-376d-46c4-9ac5-85733a50439a-tmp-dir\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.847653 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.847633 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4c07138-376d-46c4-9ac5-85733a50439a-config-volume\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.861264 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.861239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:17.866009 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.865985 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf6x8\" (UniqueName: \"kubernetes.io/projected/d4c07138-376d-46c4-9ac5-85733a50439a-kube-api-access-pf6x8\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:17.879843 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.879820 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fmtjs" Jun 15 05:49:17.949870 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:17.949840 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:18.057297 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:18.057136 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fmtjs"] Jun 15 05:49:18.059263 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:18.059240 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312ffed8_5109_4ae0_9b62_74f951fc8068.slice/crio-343f4c3c1e98c5b384692d39223bcf066329a6ba5c7e997d794fdb31d0de9044 WatchSource:0}: Error finding container 343f4c3c1e98c5b384692d39223bcf066329a6ba5c7e997d794fdb31d0de9044: Status 404 returned error can't find the container with id 343f4c3c1e98c5b384692d39223bcf066329a6ba5c7e997d794fdb31d0de9044 Jun 15 05:49:18.278081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:18.277984 2569 generic.go:356] "Generic (PLEG): container finished" podID="a834d69b-c741-447e-b35c-f303b03179cf" containerID="89c571119ae715e57a21b58c2fe968542bb32e2262a011ae093a6c90dad96b29" exitCode=0 Jun 15 05:49:18.278081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:18.278062 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerDied","Data":"89c571119ae715e57a21b58c2fe968542bb32e2262a011ae093a6c90dad96b29"} Jun 15 05:49:18.279222 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:18.279197 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmtjs" event={"ID":"312ffed8-5109-4ae0-9b62-74f951fc8068","Type":"ContainerStarted","Data":"343f4c3c1e98c5b384692d39223bcf066329a6ba5c7e997d794fdb31d0de9044"} Jun 15 05:49:18.350795 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:18.350750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:18.351037 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:18.350927 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 15 05:49:18.351037 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:18.350999 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls podName:d4c07138-376d-46c4-9ac5-85733a50439a nodeName:}" failed. No retries permitted until 2026-06-15 05:49:19.350979214 +0000 UTC m=+37.696182337 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls") pod "dns-default-d97j4" (UID: "d4c07138-376d-46c4-9ac5-85733a50439a") : secret "dns-default-metrics-tls" not found Jun 15 05:49:19.051375 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.051333 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:19.051558 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.051339 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:19.051558 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.051338 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:19.055038 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.055006 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqf9b\"" Jun 15 05:49:19.055305 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.055284 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kxrkb\"" Jun 15 05:49:19.055305 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.055296 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 15 05:49:19.055465 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.055331 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 15 05:49:19.055465 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.055328 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 15 05:49:19.055465 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.055376 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 15 05:49:19.286218 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.286186 2569 generic.go:356] "Generic (PLEG): container finished" podID="a834d69b-c741-447e-b35c-f303b03179cf" containerID="b0110a40f7608f2479c14282870c0e30a07e1d70fff20163fa163ad018bf3bdf" exitCode=0 Jun 15 05:49:19.286663 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.286228 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerDied","Data":"b0110a40f7608f2479c14282870c0e30a07e1d70fff20163fa163ad018bf3bdf"} Jun 15 05:49:19.360233 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.360148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:19.360407 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:19.360340 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 15 05:49:19.360464 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:19.360414 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls podName:d4c07138-376d-46c4-9ac5-85733a50439a nodeName:}" failed. No retries permitted until 2026-06-15 05:49:21.360391812 +0000 UTC m=+39.705594937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls") pod "dns-default-d97j4" (UID: "d4c07138-376d-46c4-9ac5-85733a50439a") : secret "dns-default-metrics-tls" not found Jun 15 05:49:19.746007 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.745931 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:19.803761 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:19.803730 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:20.292142 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:20.292111 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v2kln" event={"ID":"a834d69b-c741-447e-b35c-f303b03179cf","Type":"ContainerStarted","Data":"23bf0cad932b2f3472e76fa4ea09d13d155edacb7fd59581d3c409038d52529a"} Jun 15 05:49:20.293620 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:20.293596 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fmtjs" event={"ID":"312ffed8-5109-4ae0-9b62-74f951fc8068","Type":"ContainerStarted","Data":"d21d45aa94a1d7d1b5dcd540d057767132e2d451a10fcd62ec1f64a374344769"} Jun 15 05:49:20.328516 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:20.328471 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v2kln" podStartSLOduration=8.495251705 podStartE2EDuration="38.328455692s" totalImagesPullingTime="29.833203987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:20.328358694 +0000 UTC m=+38.673561834" watchObservedRunningTime="2026-06-15 05:49:20.328455692 +0000 UTC m=+38.673658833" Jun 15 05:49:20.357550 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:20.357463 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fmtjs" podStartSLOduration=1.962366853 podStartE2EDuration="3.357441194s" totalImagesPullingTime="1.395074341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:20.356207763 +0000 UTC m=+38.701410905" watchObservedRunningTime="2026-06-15 05:49:20.357441194 +0000 UTC m=+38.702644336" Jun 15 05:49:20.470565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:20.470517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:20.521691 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:20.521654 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:20.978124 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:20.978094 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:21.033498 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.033470 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:21.375888 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.375858 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:21.376391 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:21.376008 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 15 05:49:21.376391 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:21.376081 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls podName:d4c07138-376d-46c4-9ac5-85733a50439a nodeName:}" failed. No retries permitted until 2026-06-15 05:49:25.376061241 +0000 UTC m=+43.721264361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls") pod "dns-default-d97j4" (UID: "d4c07138-376d-46c4-9ac5-85733a50439a") : secret "dns-default-metrics-tls" not found Jun 15 05:49:21.680629 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.680528 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:21.707192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.707161 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x"] Jun 15 05:49:21.709038 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.709015 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" Jun 15 05:49:21.711571 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.711533 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jun 15 05:49:21.711571 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.711563 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jun 15 05:49:21.712071 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.712054 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nsfwg\"" Jun 15 05:49:21.718507 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.718479 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x"] Jun 15 05:49:21.750380 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.750351 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:21.778855 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.778822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cscms\" (UniqueName: \"kubernetes.io/projected/eef77ce7-de11-4e1e-83c3-6393e0628087-kube-api-access-cscms\") pod \"migrator-d4bcd4867-8dt2x\" (UID: \"eef77ce7-de11-4e1e-83c3-6393e0628087\") " pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" Jun 15 05:49:21.831599 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.831562 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:21.879716 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.879679 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cscms\" (UniqueName: \"kubernetes.io/projected/eef77ce7-de11-4e1e-83c3-6393e0628087-kube-api-access-cscms\") pod \"migrator-d4bcd4867-8dt2x\" (UID: \"eef77ce7-de11-4e1e-83c3-6393e0628087\") " pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" Jun 15 05:49:21.895957 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.895931 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cscms\" (UniqueName: \"kubernetes.io/projected/eef77ce7-de11-4e1e-83c3-6393e0628087-kube-api-access-cscms\") pod \"migrator-d4bcd4867-8dt2x\" (UID: \"eef77ce7-de11-4e1e-83c3-6393e0628087\") " pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" Jun 15 05:49:21.902234 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.902205 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:21.955370 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:21.955294 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:22.017728 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.017696 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" Jun 15 05:49:22.139648 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.139617 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x"] Jun 15 05:49:22.142930 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:22.142900 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef77ce7_de11_4e1e_83c3_6393e0628087.slice/crio-9aa1d23e7320e5bbe78b29783fb3478d7e4108d0d35cb0a544e6a6044fad0446 WatchSource:0}: Error finding container 9aa1d23e7320e5bbe78b29783fb3478d7e4108d0d35cb0a544e6a6044fad0446: Status 404 returned error can't find the container with id 9aa1d23e7320e5bbe78b29783fb3478d7e4108d0d35cb0a544e6a6044fad0446 Jun 15 05:49:22.299114 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.299086 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" event={"ID":"eef77ce7-de11-4e1e-83c3-6393e0628087","Type":"ContainerStarted","Data":"9aa1d23e7320e5bbe78b29783fb3478d7e4108d0d35cb0a544e6a6044fad0446"} Jun 15 05:49:22.731058 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.730974 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bx7hl"] Jun 15 05:49:22.734121 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.734094 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.736524 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.736393 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jun 15 05:49:22.736524 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.736473 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jun 15 05:49:22.736524 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.736492 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jun 15 05:49:22.736762 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.736629 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jun 15 05:49:22.737531 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.737512 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lbvfg\"" Jun 15 05:49:22.746156 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.746137 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bx7hl"] Jun 15 05:49:22.753342 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.753313 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:22.814696 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.814662 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:22.888403 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.888370 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/914454af-bf9e-49b0-bfbb-e0d5d551d834-data-volume\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.888403 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.888408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.888626 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.888427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.888626 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.888446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/914454af-bf9e-49b0-bfbb-e0d5d551d834-crio-socket\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.888626 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.888465 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89qs9\" (UniqueName: \"kubernetes.io/projected/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-api-access-89qs9\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.989616 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.989522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/914454af-bf9e-49b0-bfbb-e0d5d551d834-data-volume\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.989616 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.989579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.989821 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:22.989644 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 15 05:49:22.989821 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.989695 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.989821 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:22.989713 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:23.489692768 +0000 UTC m=+41.834895887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : configmap "kube-rbac-proxy" not found Jun 15 05:49:22.989821 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.989765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/914454af-bf9e-49b0-bfbb-e0d5d551d834-crio-socket\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.989821 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.989799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89qs9\" (UniqueName: \"kubernetes.io/projected/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-api-access-89qs9\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.989821 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:22.989820 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 15 05:49:22.990059 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:22.989869 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:23.489852244 +0000 UTC m=+41.835055367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : secret "insights-runtime-extractor-tls" not found Jun 15 05:49:22.990059 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.990028 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/914454af-bf9e-49b0-bfbb-e0d5d551d834-crio-socket\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.990609 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.990590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/914454af-bf9e-49b0-bfbb-e0d5d551d834-data-volume\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:22.997753 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:22.997730 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89qs9\" (UniqueName: \"kubernetes.io/projected/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-api-access-89qs9\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:23.303005 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.302971 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" event={"ID":"eef77ce7-de11-4e1e-83c3-6393e0628087","Type":"ContainerStarted","Data":"bf65b21317abb9722143e84e689119e322a611a95fb90f83b07188b8949c110b"} Jun 15 05:49:23.303155 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.303048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" event={"ID":"eef77ce7-de11-4e1e-83c3-6393e0628087","Type":"ContainerStarted","Data":"78b5430562b6530357c03bf1177449bfcd6ce17c8fdb3af244746af7d853a35a"} Jun 15 05:49:23.319165 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.319121 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-8dt2x" podStartSLOduration=1.300797084 podStartE2EDuration="2.319101581s" totalImagesPullingTime="1.018304497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:23.31899327 +0000 UTC m=+41.664196411" watchObservedRunningTime="2026-06-15 05:49:23.319101581 +0000 UTC m=+41.664304722" Jun 15 05:49:23.409051 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.409013 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:23.493965 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.493927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:23.493965 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.493965 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:23.494185 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:23.494059 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 15 05:49:23.494185 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:23.494080 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 15 05:49:23.494185 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:23.494136 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:24.494113352 +0000 UTC m=+42.839316486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : configmap "kube-rbac-proxy" not found Jun 15 05:49:23.494185 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:23.494159 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:24.494147704 +0000 UTC m=+42.839350826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : secret "insights-runtime-extractor-tls" not found Jun 15 05:49:23.509586 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.509506 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:23.609335 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.609304 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:23.705828 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:23.705798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:24.042514 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:24.042478 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:24.143527 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:24.143498 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:24.320139 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:24.320065 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:24.419361 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:24.419333 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:24.503329 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:24.503291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:24.503329 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:24.503329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:24.503533 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:24.503408 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 15 05:49:24.503533 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:24.503473 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:26.503458237 +0000 UTC m=+44.848661361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : configmap "kube-rbac-proxy" not found Jun 15 05:49:24.503533 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:24.503491 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 15 05:49:24.503658 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:24.503535 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:26.503520101 +0000 UTC m=+44.848723221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : secret "insights-runtime-extractor-tls" not found Jun 15 05:49:25.079622 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.079587 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:25.180827 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.180795 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:25.411203 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.411106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:25.411353 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:25.411254 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jun 15 05:49:25.411353 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:25.411320 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls podName:d4c07138-376d-46c4-9ac5-85733a50439a nodeName:}" failed. No retries permitted until 2026-06-15 05:49:33.411303708 +0000 UTC m=+51.756506828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls") pod "dns-default-d97j4" (UID: "d4c07138-376d-46c4-9ac5-85733a50439a") : secret "dns-default-metrics-tls" not found Jun 15 05:49:25.704851 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.704781 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqzpb_a19a3731-5372-4d37-83aa-6474aad11f0e/dns-node-resolver/0.log" Jun 15 05:49:25.919692 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.919659 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-nstd9"] Jun 15 05:49:25.961470 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.961390 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:25.996014 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.995988 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-nstd9"] Jun 15 05:49:25.996130 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.996087 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:25.998806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.998785 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6vmkf\"" Jun 15 05:49:25.998936 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.998805 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jun 15 05:49:25.999468 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.999452 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jun 15 05:49:25.999629 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.999614 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jun 15 05:49:25.999699 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:25.999616 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jun 15 05:49:26.015299 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.015272 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:26.116057 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.116006 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95e2e585-231c-4a67-825e-d6ecc88acf55-signing-cabundle\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.116423 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.116071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95e2e585-231c-4a67-825e-d6ecc88acf55-signing-key\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.116423 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.116096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rbb\" (UniqueName: \"kubernetes.io/projected/95e2e585-231c-4a67-825e-d6ecc88acf55-kube-api-access-w2rbb\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.217367 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.217283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95e2e585-231c-4a67-825e-d6ecc88acf55-signing-cabundle\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.217367 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.217333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95e2e585-231c-4a67-825e-d6ecc88acf55-signing-key\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.217367 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.217355 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rbb\" (UniqueName: \"kubernetes.io/projected/95e2e585-231c-4a67-825e-d6ecc88acf55-kube-api-access-w2rbb\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.218143 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.218112 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/95e2e585-231c-4a67-825e-d6ecc88acf55-signing-cabundle\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.219744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.219716 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/95e2e585-231c-4a67-825e-d6ecc88acf55-signing-key\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.227625 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.227606 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rbb\" (UniqueName: \"kubernetes.io/projected/95e2e585-231c-4a67-825e-d6ecc88acf55-kube-api-access-w2rbb\") pod \"service-ca-7b994bc59-nstd9\" (UID: \"95e2e585-231c-4a67-825e-d6ecc88acf55\") " pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.305497 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.305467 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7b994bc59-nstd9" Jun 15 05:49:26.454898 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.454868 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-nstd9"] Jun 15 05:49:26.457765 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:26.457740 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95e2e585_231c_4a67_825e_d6ecc88acf55.slice/crio-e41247e70d611f3cf1e0fecb737e65d3675d62d85995a8c2e47a5098684e66f2 WatchSource:0}: Error finding container e41247e70d611f3cf1e0fecb737e65d3675d62d85995a8c2e47a5098684e66f2: Status 404 returned error can't find the container with id e41247e70d611f3cf1e0fecb737e65d3675d62d85995a8c2e47a5098684e66f2 Jun 15 05:49:26.505334 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.505264 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g28pn_2be932e7-3cd4-48b1-88ac-e9cead0e0a1e/node-ca/0.log" Jun 15 05:49:26.519677 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.519645 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:26.519797 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.519686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:26.519797 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:26.519761 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jun 15 05:49:26.519797 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:26.519785 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 15 05:49:26.519911 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:26.519818 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:30.519801988 +0000 UTC m=+48.865005107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : configmap "kube-rbac-proxy" not found Jun 15 05:49:26.519911 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:26.519832 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:30.519825721 +0000 UTC m=+48.865028840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : secret "insights-runtime-extractor-tls" not found Jun 15 05:49:26.784626 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:26.784592 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:27.313892 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:27.313852 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7b994bc59-nstd9" event={"ID":"95e2e585-231c-4a67-825e-d6ecc88acf55","Type":"ContainerStarted","Data":"e41247e70d611f3cf1e0fecb737e65d3675d62d85995a8c2e47a5098684e66f2"} Jun 15 05:49:27.504432 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:27.504404 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fmtjs_312ffed8-5109-4ae0-9b62-74f951fc8068/serve-healthcheck-canary/0.log" Jun 15 05:49:27.704756 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:27.704692 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-8dt2x_eef77ce7-de11-4e1e-83c3-6393e0628087/migrator/0.log" Jun 15 05:49:27.908199 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:27.908170 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-8dt2x_eef77ce7-de11-4e1e-83c3-6393e0628087/graceful-termination/0.log" Jun 15 05:49:28.012358 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:28.012272 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:29.251343 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:29.251318 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:29.320503 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:29.320469 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7b994bc59-nstd9" event={"ID":"95e2e585-231c-4a67-825e-d6ecc88acf55","Type":"ContainerStarted","Data":"6733a169c5d68a1129c8f7c8776e38e9997163470a86431893ee01bb9584c2f7"} Jun 15 05:49:29.343832 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:29.343756 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7b994bc59-nstd9" podStartSLOduration=1.612555647 podStartE2EDuration="4.343738749s" totalImagesPullingTime="2.731183102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:29.342915048 +0000 UTC m=+47.688118186" watchObservedRunningTime="2026-06-15 05:49:29.343738749 +0000 UTC m=+47.688941919" Jun 15 05:49:30.228306 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.228270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:30.493557 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.493421 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:30.552026 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.551984 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:30.552183 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:30.552140 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jun 15 05:49:30.552252 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:30.552214 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls podName:914454af-bf9e-49b0-bfbb-e0d5d551d834 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:38.552193249 +0000 UTC m=+56.897396378 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bx7hl" (UID: "914454af-bf9e-49b0-bfbb-e0d5d551d834") : secret "insights-runtime-extractor-tls" not found Jun 15 05:49:30.552567 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.552146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:30.552851 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.552818 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/914454af-bf9e-49b0-bfbb-e0d5d551d834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:30.595037 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.595000 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:30.699031 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.699002 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:30.929127 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:30.929097 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:31.724882 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:31.724844 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:32.326584 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:32.326524 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:32.529485 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:32.529449 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:32.721140 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:32.721032 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:33.125771 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.125741 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:33.477854 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.477761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:33.480131 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.480110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4c07138-376d-46c4-9ac5-85733a50439a-metrics-tls\") pod \"dns-default-d97j4\" (UID: \"d4c07138-376d-46c4-9ac5-85733a50439a\") " pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:33.503244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.503217 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:33.523523 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.523487 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:33.624351 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.624323 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:33.651825 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.651784 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d97j4"] Jun 15 05:49:33.654597 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:33.654565 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c07138_376d_46c4_9ac5_85733a50439a.slice/crio-8bd6f6977019d7ceba63d2d983861cdbaf47d0342601716dbeebd1c6f0d1507d WatchSource:0}: Error finding container 8bd6f6977019d7ceba63d2d983861cdbaf47d0342601716dbeebd1c6f0d1507d: Status 404 returned error can't find the container with id 8bd6f6977019d7ceba63d2d983861cdbaf47d0342601716dbeebd1c6f0d1507d Jun 15 05:49:33.724399 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.724366 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:33.931157 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:33.931125 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:34.126320 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:34.126284 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:34.339238 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:34.339193 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d97j4" event={"ID":"d4c07138-376d-46c4-9ac5-85733a50439a","Type":"ContainerStarted","Data":"8bd6f6977019d7ceba63d2d983861cdbaf47d0342601716dbeebd1c6f0d1507d"} Jun 15 05:49:34.529866 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:34.529833 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:34.582841 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:34.582810 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:34.730142 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:34.730064 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:35.324690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:35.324650 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:35.926651 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:35.926533 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:36.125769 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.125726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:36.227852 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.227809 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:36.327306 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.327270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:36.348160 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.348120 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d97j4" event={"ID":"d4c07138-376d-46c4-9ac5-85733a50439a","Type":"ContainerStarted","Data":"51738b77537d441c69a2b22b0f9c7e9c170d44c3d38e5beb3e258d9686cf5482"} Jun 15 05:49:36.348331 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.348171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d97j4" event={"ID":"d4c07138-376d-46c4-9ac5-85733a50439a","Type":"ContainerStarted","Data":"261569d4821ad66345a0b4c68cc354ff5e5de9871d8e9cafd526ce1b4e1d01bc"} Jun 15 05:49:36.348331 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.348255 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:36.366492 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.366449 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d97j4" podStartSLOduration=17.279819998 podStartE2EDuration="19.366434141s" totalImagesPullingTime="2.086614143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:36.365317378 +0000 UTC m=+54.710520522" watchObservedRunningTime="2026-06-15 05:49:36.366434141 +0000 UTC m=+54.711637281" Jun 15 05:49:36.521895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.521863 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:36.730718 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:36.730634 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:37.135473 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:37.135442 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:37.527215 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:37.527123 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:38.126765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:38.126726 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:38.526156 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:38.526069 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:38.619691 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:38.619657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:38.621982 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:38.621959 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/914454af-bf9e-49b0-bfbb-e0d5d551d834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bx7hl\" (UID: \"914454af-bf9e-49b0-bfbb-e0d5d551d834\") " pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:38.646126 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:38.646071 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bx7hl" Jun 15 05:49:38.776035 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:38.776004 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bx7hl"] Jun 15 05:49:38.778912 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:38.778879 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914454af_bf9e_49b0_bfbb_e0d5d551d834.slice/crio-37914080776f57b40fa3604c06420446d1ba45f38e2db0619f0640d9a7aab466 WatchSource:0}: Error finding container 37914080776f57b40fa3604c06420446d1ba45f38e2db0619f0640d9a7aab466: Status 404 returned error can't find the container with id 37914080776f57b40fa3604c06420446d1ba45f38e2db0619f0640d9a7aab466 Jun 15 05:49:38.928743 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:38.928709 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:39.127910 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:39.127869 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:39.371066 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:39.371028 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bx7hl" event={"ID":"914454af-bf9e-49b0-bfbb-e0d5d551d834","Type":"ContainerStarted","Data":"37914080776f57b40fa3604c06420446d1ba45f38e2db0619f0640d9a7aab466"} Jun 15 05:49:39.524338 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:39.524230 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:40.084092 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:40.084053 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:40.185042 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:40.185013 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:40.339044 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:40.338965 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:40.525981 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:40.525941 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:40.730808 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:40.730704 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:40.924136 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:40.924100 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:42.389784 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:42.389692 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bx7hl" event={"ID":"914454af-bf9e-49b0-bfbb-e0d5d551d834","Type":"ContainerStarted","Data":"554fa822ac225c71001f7d7db2460992a8b1ed588b9e834b17e7801f57394cc2"} Jun 15 05:49:43.278764 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:43.278739 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rkdkp" Jun 15 05:49:43.393944 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:43.393908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bx7hl" event={"ID":"914454af-bf9e-49b0-bfbb-e0d5d551d834","Type":"ContainerStarted","Data":"c8eeda9c6a1772f4d3d16acf59797447df38a40f4c89a1139a1a0865a9f5b32f"} Jun 15 05:49:45.403049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:45.403020 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bx7hl" event={"ID":"914454af-bf9e-49b0-bfbb-e0d5d551d834","Type":"ContainerStarted","Data":"c2994ff5ce8fb89e9b6b68278fd249fef54b1d4838310f153d0ed14f34f6fafe"} Jun 15 05:49:46.353779 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.353745 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d97j4" Jun 15 05:49:46.374665 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.374612 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bx7hl" podStartSLOduration=18.36086681 podStartE2EDuration="24.374596953s" totalImagesPullingTime="6.013730143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:49:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:45.421496339 +0000 UTC m=+63.766699474" watchObservedRunningTime="2026-06-15 05:49:46.374596953 +0000 UTC m=+64.719800094" Jun 15 05:49:46.626475 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.626397 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:46.776744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.776691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:46.780244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.780223 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jun 15 05:49:46.789567 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.789530 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9039cf21-b4dc-4e30-bdb6-a08444213d19-original-pull-secret\") pod \"global-pull-secret-syncer-pvj8k\" (UID: \"9039cf21-b4dc-4e30-bdb6-a08444213d19\") " pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:46.878018 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.877932 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:46.880976 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.880951 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jun 15 05:49:46.891208 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.891178 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad21e817-c798-46ed-a424-d9ac7ea41cb7-metrics-certs\") pod \"network-metrics-daemon-pxmpj\" (UID: \"ad21e817-c798-46ed-a424-d9ac7ea41cb7\") " pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:46.970909 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.970864 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvj8k" Jun 15 05:49:46.979123 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.979099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:46.979517 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.979499 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tqf9b\"" Jun 15 05:49:46.981846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.981819 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jun 15 05:49:46.987802 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.987785 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pxmpj" Jun 15 05:49:46.991965 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:46.991946 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jun 15 05:49:47.002258 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.002235 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2tbj\" (UniqueName: \"kubernetes.io/projected/e8db85b7-db76-4d90-830a-af7729764788-kube-api-access-h2tbj\") pod \"network-check-target-8n9hm\" (UID: \"e8db85b7-db76-4d90-830a-af7729764788\") " pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:47.106392 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.106360 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvj8k"] Jun 15 05:49:47.110341 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:47.110315 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9039cf21_b4dc_4e30_bdb6_a08444213d19.slice/crio-f0de3bafecf65daa01b08a59b85e0bfc321c7986f54107c1fae4ce2a32204a3b WatchSource:0}: Error finding container f0de3bafecf65daa01b08a59b85e0bfc321c7986f54107c1fae4ce2a32204a3b: Status 404 returned error can't find the container with id f0de3bafecf65daa01b08a59b85e0bfc321c7986f54107c1fae4ce2a32204a3b Jun 15 05:49:47.123828 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.123801 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pxmpj"] Jun 15 05:49:47.126983 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:47.126960 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad21e817_c798_46ed_a424_d9ac7ea41cb7.slice/crio-119eba2bad6ae80ef9deb6ad891379122d7c42336e11c62f219acce3a4c03d8a WatchSource:0}: Error finding container 119eba2bad6ae80ef9deb6ad891379122d7c42336e11c62f219acce3a4c03d8a: Status 404 returned error can't find the container with id 119eba2bad6ae80ef9deb6ad891379122d7c42336e11c62f219acce3a4c03d8a Jun 15 05:49:47.265692 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.265661 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-kxrkb\"" Jun 15 05:49:47.273924 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.273895 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:47.398909 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.398824 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8n9hm"] Jun 15 05:49:47.401785 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:47.401757 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8db85b7_db76_4d90_830a_af7729764788.slice/crio-8cf4ef5c6ec6920c1cdb0085293ebc2e7f2b7a62093cd0bf9b58c8c914b2d35d WatchSource:0}: Error finding container 8cf4ef5c6ec6920c1cdb0085293ebc2e7f2b7a62093cd0bf9b58c8c914b2d35d: Status 404 returned error can't find the container with id 8cf4ef5c6ec6920c1cdb0085293ebc2e7f2b7a62093cd0bf9b58c8c914b2d35d Jun 15 05:49:47.410459 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.410429 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvj8k" event={"ID":"9039cf21-b4dc-4e30-bdb6-a08444213d19","Type":"ContainerStarted","Data":"f0de3bafecf65daa01b08a59b85e0bfc321c7986f54107c1fae4ce2a32204a3b"} Jun 15 05:49:47.411349 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.411324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8n9hm" event={"ID":"e8db85b7-db76-4d90-830a-af7729764788","Type":"ContainerStarted","Data":"8cf4ef5c6ec6920c1cdb0085293ebc2e7f2b7a62093cd0bf9b58c8c914b2d35d"} Jun 15 05:49:47.412288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:47.412267 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxmpj" event={"ID":"ad21e817-c798-46ed-a424-d9ac7ea41cb7","Type":"ContainerStarted","Data":"119eba2bad6ae80ef9deb6ad891379122d7c42336e11c62f219acce3a4c03d8a"} Jun 15 05:49:48.420253 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:48.419948 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxmpj" event={"ID":"ad21e817-c798-46ed-a424-d9ac7ea41cb7","Type":"ContainerStarted","Data":"2ee6b126c8022e2bd496b5b45daab5f77a38d8f78d4db9395a94d81a723ccd28"} Jun 15 05:49:48.420253 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:48.420016 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pxmpj" event={"ID":"ad21e817-c798-46ed-a424-d9ac7ea41cb7","Type":"ContainerStarted","Data":"34e5fed2217e9f2eff7090fb630a7af5aa57f04472add6eb5af3ba7e1cdf4351"} Jun 15 05:49:48.447001 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:48.444807 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pxmpj" podStartSLOduration=65.596834349 podStartE2EDuration="1m6.444785154s" totalImagesPullingTime="847.950805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:48.442352472 +0000 UTC m=+66.787555625" watchObservedRunningTime="2026-06-15 05:49:48.444785154 +0000 UTC m=+66.789988298" Jun 15 05:49:48.633237 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:48.633202 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:48.729258 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:48.729142 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:48.834914 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:48.834875 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:48.929347 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:48.929309 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:50.918784 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:50.918744 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:51.001047 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.001010 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:51.099920 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.099891 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:51.103171 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.103142 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk"] Jun 15 05:49:51.118651 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.117990 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk"] Jun 15 05:49:51.118651 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.118091 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:51.120881 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.120858 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-djh4c\"" Jun 15 05:49:51.120881 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.120876 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jun 15 05:49:51.173008 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.172920 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:51.213039 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.212982 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21666edb-c19b-470b-9ae8-75b6dee41a50-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-fkznk\" (UID: \"21666edb-c19b-470b-9ae8-75b6dee41a50\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:51.313817 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.313783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21666edb-c19b-470b-9ae8-75b6dee41a50-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-fkznk\" (UID: \"21666edb-c19b-470b-9ae8-75b6dee41a50\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:51.314015 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:51.313947 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jun 15 05:49:51.314076 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:51.314029 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21666edb-c19b-470b-9ae8-75b6dee41a50-tls-certificates podName:21666edb-c19b-470b-9ae8-75b6dee41a50 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:51.814005468 +0000 UTC m=+70.159208593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/21666edb-c19b-470b-9ae8-75b6dee41a50-tls-certificates") pod "prometheus-operator-admission-webhook-5cf667749c-fkznk" (UID: "21666edb-c19b-470b-9ae8-75b6dee41a50") : secret "prometheus-operator-admission-webhook-tls" not found Jun 15 05:49:51.506853 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.506769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:51.817498 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.817454 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21666edb-c19b-470b-9ae8-75b6dee41a50-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-fkznk\" (UID: \"21666edb-c19b-470b-9ae8-75b6dee41a50\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:51.820279 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:51.820251 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/21666edb-c19b-470b-9ae8-75b6dee41a50-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-fkznk\" (UID: \"21666edb-c19b-470b-9ae8-75b6dee41a50\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:52.029233 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.029196 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:52.313028 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.312995 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk"] Jun 15 05:49:52.315915 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:52.315888 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21666edb_c19b_470b_9ae8_75b6dee41a50.slice/crio-68029b716e35bef0e09053c0c7461552c94d256f63c061022643f7e38ace54f1 WatchSource:0}: Error finding container 68029b716e35bef0e09053c0c7461552c94d256f63c061022643f7e38ace54f1: Status 404 returned error can't find the container with id 68029b716e35bef0e09053c0c7461552c94d256f63c061022643f7e38ace54f1 Jun 15 05:49:52.439960 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.439870 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8n9hm" event={"ID":"e8db85b7-db76-4d90-830a-af7729764788","Type":"ContainerStarted","Data":"d6dd0d7b73da2de2e38888d8dbe3f0aca084904dcb5bd845436e398a853bcfa7"} Jun 15 05:49:52.439960 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.439951 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:49:52.441070 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.441048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" event={"ID":"21666edb-c19b-470b-9ae8-75b6dee41a50","Type":"ContainerStarted","Data":"68029b716e35bef0e09053c0c7461552c94d256f63c061022643f7e38ace54f1"} Jun 15 05:49:52.442377 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.442360 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvj8k" event={"ID":"9039cf21-b4dc-4e30-bdb6-a08444213d19","Type":"ContainerStarted","Data":"57467dc84ea020331da7687043ed6ae921ed65777a0a316fd51385ba7e987528"} Jun 15 05:49:52.460374 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.460235 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8n9hm" podStartSLOduration=65.676074713 podStartE2EDuration="1m10.460222629s" totalImagesPullingTime="4.784147916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:52.459863101 +0000 UTC m=+70.805066244" watchObservedRunningTime="2026-06-15 05:49:52.460222629 +0000 UTC m=+70.805425804" Jun 15 05:49:52.476085 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:52.476039 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pvj8k" podStartSLOduration=65.404460109 podStartE2EDuration="1m10.476024766s" totalImagesPullingTime="5.071564657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:52.475515115 +0000 UTC m=+70.820718267" watchObservedRunningTime="2026-06-15 05:49:52.476024766 +0000 UTC m=+70.821227906" Jun 15 05:49:53.447084 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:53.447039 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" event={"ID":"21666edb-c19b-470b-9ae8-75b6dee41a50","Type":"ContainerStarted","Data":"2888f6d817c4b5a1fd1be2dc77f343ede231f6f4177bcc701dcc0428c51181e5"} Jun 15 05:49:54.047053 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:54.047006 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:54.146475 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:54.146427 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:54.450169 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:54.450084 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:54.454806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:54.454780 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" Jun 15 05:49:54.480664 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:54.480615 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-fkznk" podStartSLOduration=2.501053252 podStartE2EDuration="3.480598753s" totalImagesPullingTime="979.545501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:53.469444353 +0000 UTC m=+71.814647497" watchObservedRunningTime="2026-06-15 05:49:54.480598753 +0000 UTC m=+72.825801895" Jun 15 05:49:55.109974 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.109919 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:55.196700 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.196666 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm"] Jun 15 05:49:55.198713 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.198696 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.200001 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.199972 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:55.201158 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.201142 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jun 15 05:49:55.201158 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.201154 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jun 15 05:49:55.201923 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.201908 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jun 15 05:49:55.202244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.202227 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jun 15 05:49:55.202407 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.202394 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jun 15 05:49:55.202491 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.202431 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-s6gmg\"" Jun 15 05:49:55.211689 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.211661 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm"] Jun 15 05:49:55.245690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.245657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-metrics-client-ca\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.245841 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.245713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7lvm\" (UniqueName: \"kubernetes.io/projected/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-kube-api-access-t7lvm\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.245841 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.245733 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.245841 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.245753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.346627 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.346579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7lvm\" (UniqueName: \"kubernetes.io/projected/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-kube-api-access-t7lvm\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.346627 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.346623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.346825 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.346647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.346825 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:55.346745 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jun 15 05:49:55.346825 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:49:55.346804 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-tls podName:144f3ad2-f47e-47aa-b482-7f4d0fbd473f nodeName:}" failed. No retries permitted until 2026-06-15 05:49:55.846785909 +0000 UTC m=+74.191989047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-tls") pod "prometheus-operator-864fcfc6f9-jdntm" (UID: "144f3ad2-f47e-47aa-b482-7f4d0fbd473f") : secret "prometheus-operator-tls" not found Jun 15 05:49:55.346925 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.346826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-metrics-client-ca\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.347448 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.347420 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-metrics-client-ca\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.348899 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.348877 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.357953 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.357928 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7lvm\" (UniqueName: \"kubernetes.io/projected/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-kube-api-access-t7lvm\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.643432 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.643393 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:55.850709 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.850671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:55.853097 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:55.853060 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/144f3ad2-f47e-47aa-b482-7f4d0fbd473f-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-jdntm\" (UID: \"144f3ad2-f47e-47aa-b482-7f4d0fbd473f\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:56.109872 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:56.109835 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" Jun 15 05:49:56.237608 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:56.237578 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm"] Jun 15 05:49:56.240686 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:49:56.240660 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144f3ad2_f47e_47aa_b482_7f4d0fbd473f.slice/crio-64c543e08a645f47e870afb4890503607257d0549a38d0c65b5a37488fe1979a WatchSource:0}: Error finding container 64c543e08a645f47e870afb4890503607257d0549a38d0c65b5a37488fe1979a: Status 404 returned error can't find the container with id 64c543e08a645f47e870afb4890503607257d0549a38d0c65b5a37488fe1979a Jun 15 05:49:56.458674 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:56.458589 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" event={"ID":"144f3ad2-f47e-47aa-b482-7f4d0fbd473f","Type":"ContainerStarted","Data":"64c543e08a645f47e870afb4890503607257d0549a38d0c65b5a37488fe1979a"} Jun 15 05:49:57.644794 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:57.644769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:57.744755 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:57.744722 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:57.847790 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:57.847701 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:49:58.467170 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:58.467132 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" event={"ID":"144f3ad2-f47e-47aa-b482-7f4d0fbd473f","Type":"ContainerStarted","Data":"062949f1b934006c6566ba6e7c962e943ceed4d8bda01f655eb6a3666e24111f"} Jun 15 05:49:58.467380 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:58.467183 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" event={"ID":"144f3ad2-f47e-47aa-b482-7f4d0fbd473f","Type":"ContainerStarted","Data":"aee2ea8b86f41a05f0da15b7f081b7299dbbbc76425e879a76855454a34f6f51"} Jun 15 05:49:58.486144 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:49:58.486098 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-jdntm" podStartSLOduration=2.196518502 podStartE2EDuration="3.486082947s" totalImagesPullingTime="1.289564445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:58.485245937 +0000 UTC m=+76.830449077" watchObservedRunningTime="2026-06-15 05:49:58.486082947 +0000 UTC m=+76.831286088" Jun 15 05:50:00.316245 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.316210 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:00.583443 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.583352 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:00.684807 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.684767 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:00.701979 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.701946 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2k5fq"] Jun 15 05:50:00.705770 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.705748 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.708297 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.708272 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jun 15 05:50:00.708743 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.708723 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jun 15 05:50:00.708979 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.708955 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jun 15 05:50:00.709080 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.708958 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzg2h\"" Jun 15 05:50:00.720085 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.720066 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l"] Jun 15 05:50:00.722890 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.722874 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.725070 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.725043 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jun 15 05:50:00.725162 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.725117 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-v4mcb\"" Jun 15 05:50:00.725609 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.725592 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jun 15 05:50:00.725885 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.725870 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jun 15 05:50:00.735657 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.735629 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l"] Jun 15 05:50:00.765689 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.765653 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:00.784843 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.784810 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-textfile\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785022 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.784875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfhb\" (UniqueName: \"kubernetes.io/projected/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-api-access-hkfhb\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.785022 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.784929 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785022 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.784961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.785022 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.784990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785022 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785011 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-sys\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785264 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785071 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-root\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785264 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d190abf6-db4f-42e4-a063-87d280b4e39e-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.785264 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-tls\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785264 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785230 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.785405 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhs5z\" (UniqueName: \"kubernetes.io/projected/fd928336-049e-4cca-88a1-bfdeb087f7f9-kube-api-access-hhs5z\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785405 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-wtmp\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785405 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd928336-049e-4cca-88a1-bfdeb087f7f9-metrics-client-ca\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.785507 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d190abf6-db4f-42e4-a063-87d280b4e39e-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.785507 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.785425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.866078 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.865930 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:00.885954 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.885920 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.885954 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.885957 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhs5z\" (UniqueName: \"kubernetes.io/projected/fd928336-049e-4cca-88a1-bfdeb087f7f9-kube-api-access-hhs5z\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.885976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-wtmp\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.885998 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd928336-049e-4cca-88a1-bfdeb087f7f9-metrics-client-ca\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d190abf6-db4f-42e4-a063-87d280b4e39e-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.886192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.886192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886158 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-wtmp\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-textfile\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886219 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfhb\" (UniqueName: \"kubernetes.io/projected/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-api-access-hkfhb\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.886490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886251 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886297 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.886490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886367 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-sys\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886447 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-root\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.886490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886482 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d190abf6-db4f-42e4-a063-87d280b4e39e-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886497 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-textfile\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-tls\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886585 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-sys\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886665 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd928336-049e-4cca-88a1-bfdeb087f7f9-metrics-client-ca\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886751 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd928336-049e-4cca-88a1-bfdeb087f7f9-root\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.886944 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d190abf6-db4f-42e4-a063-87d280b4e39e-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:50:00.887075 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:50:00.887131 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-tls podName:fd928336-049e-4cca-88a1-bfdeb087f7f9 nodeName:}" failed. No retries permitted until 2026-06-15 05:50:01.387112266 +0000 UTC m=+79.732315391 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-tls") pod "node-exporter-2k5fq" (UID: "fd928336-049e-4cca-88a1-bfdeb087f7f9") : secret "node-exporter-tls" not found Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.887165 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.887176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d190abf6-db4f-42e4-a063-87d280b4e39e-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.887395 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.887240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-accelerators-collector-config\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.888921 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.888887 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.889119 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.889099 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.889635 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.889605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.897641 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.897618 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfhb\" (UniqueName: \"kubernetes.io/projected/d190abf6-db4f-42e4-a063-87d280b4e39e-kube-api-access-hkfhb\") pod \"kube-state-metrics-678bc549d7-9xm6l\" (UID: \"d190abf6-db4f-42e4-a063-87d280b4e39e\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:00.897873 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.897855 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhs5z\" (UniqueName: \"kubernetes.io/projected/fd928336-049e-4cca-88a1-bfdeb087f7f9-kube-api-access-hhs5z\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:00.966164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:00.966131 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:01.032301 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.032267 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" Jun 15 05:50:01.069228 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.069195 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:01.169829 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.169778 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:01.170119 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.170088 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l"] Jun 15 05:50:01.174405 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:50:01.174380 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd190abf6_db4f_42e4_a063_87d280b4e39e.slice/crio-6c173d9a625405033bdeca00231143a378924fcbb39e37f6af2eb3bfeb984295 WatchSource:0}: Error finding container 6c173d9a625405033bdeca00231143a378924fcbb39e37f6af2eb3bfeb984295: Status 404 returned error can't find the container with id 6c173d9a625405033bdeca00231143a378924fcbb39e37f6af2eb3bfeb984295 Jun 15 05:50:01.379647 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.379530 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:01.389603 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.389510 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-tls\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:01.391715 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.391695 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd928336-049e-4cca-88a1-bfdeb087f7f9-node-exporter-tls\") pod \"node-exporter-2k5fq\" (UID: \"fd928336-049e-4cca-88a1-bfdeb087f7f9\") " pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:01.477810 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.477773 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" event={"ID":"d190abf6-db4f-42e4-a063-87d280b4e39e","Type":"ContainerStarted","Data":"6c173d9a625405033bdeca00231143a378924fcbb39e37f6af2eb3bfeb984295"} Jun 15 05:50:01.569604 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.569574 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:01.615705 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.615668 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2k5fq" Jun 15 05:50:01.658006 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:50:01.657981 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd928336_049e_4cca_88a1_bfdeb087f7f9.slice/crio-62b1722cd75db95749e5db434317144a821a78d605218b6849584b09bfb5bf36 WatchSource:0}: Error finding container 62b1722cd75db95749e5db434317144a821a78d605218b6849584b09bfb5bf36: Status 404 returned error can't find the container with id 62b1722cd75db95749e5db434317144a821a78d605218b6849584b09bfb5bf36 Jun 15 05:50:01.673805 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.673780 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:50:01.678489 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.678466 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:01.711506 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.711472 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:50:01.711698 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.711662 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.714629 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714591 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 15 05:50:01.714629 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714591 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 15 05:50:01.714848 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714647 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tb5vz\"" Jun 15 05:50:01.714848 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714654 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 15 05:50:01.714848 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714682 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 15 05:50:01.715018 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714880 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 15 05:50:01.715018 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714956 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 15 05:50:01.715018 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714956 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 15 05:50:01.715018 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.714975 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 15 05:50:01.715219 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.715051 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 15 05:50:01.791900 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.791860 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.791900 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.791894 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.791923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-config-out\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.791972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-web-config\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792040 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792064 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792126 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792482 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792177 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9s5h\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-kube-api-access-q9s5h\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792482 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792216 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792482 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.792482 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.792266 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893120 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893077 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893295 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893295 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893295 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893194 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893295 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893218 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893529 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893291 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-config-out\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893529 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:50:01.893357 2569 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jun 15 05:50:01.893529 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893409 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-web-config\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893529 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:50:01.893446 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls podName:5855cd37-1505-47dc-963b-a84fea2d4df1 nodeName:}" failed. No retries permitted until 2026-06-15 05:50:02.39342596 +0000 UTC m=+80.738629086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1") : secret "alertmanager-main-tls" not found Jun 15 05:50:01.893529 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893486 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893529 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893881 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893881 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893881 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.893881 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.893691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9s5h\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-kube-api-access-q9s5h\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.895267 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.895220 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.895603 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.895577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.895790 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.895765 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.897176 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.897134 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-web-config\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.897280 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.897195 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.897392 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.897331 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.897392 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.897349 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.897603 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.897458 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-config-out\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.898029 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.898001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-config-volume\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.899216 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.899192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.899651 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.899627 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.899827 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.899739 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:01.906805 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.906750 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9s5h\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-kube-api-access-q9s5h\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:01.994381 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:01.994345 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:02.301357 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:02.301314 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:02.397569 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:02.397520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:02.400259 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:02.400214 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:02.483626 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:02.483588 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k5fq" event={"ID":"fd928336-049e-4cca-88a1-bfdeb087f7f9","Type":"ContainerStarted","Data":"62b1722cd75db95749e5db434317144a821a78d605218b6849584b09bfb5bf36"} Jun 15 05:50:02.622761 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:02.622673 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:50:03.140770 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:03.140732 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:03.240605 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:03.240574 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:03.449856 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:03.449831 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:50:03.605604 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:03.605567 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:03.693814 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:03.693778 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:03.800560 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:50:03.800514 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5855cd37_1505_47dc_963b_a84fea2d4df1.slice/crio-f480011aa025cb77d43db4be40ce8c236f37d9eb897946e81ca7413fd6104fb5 WatchSource:0}: Error finding container f480011aa025cb77d43db4be40ce8c236f37d9eb897946e81ca7413fd6104fb5: Status 404 returned error can't find the container with id f480011aa025cb77d43db4be40ce8c236f37d9eb897946e81ca7413fd6104fb5 Jun 15 05:50:03.917803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:03.917762 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:04.494769 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:04.494629 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k5fq" event={"ID":"fd928336-049e-4cca-88a1-bfdeb087f7f9","Type":"ContainerStarted","Data":"abcdc33bb4f558b8e65af058310624e0fab2632b4de03109fce72b058661864b"} Jun 15 05:50:04.497406 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:04.497330 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" event={"ID":"d190abf6-db4f-42e4-a063-87d280b4e39e","Type":"ContainerStarted","Data":"814596ce31a34c984b7e197940f47f46796e752447264f63d27ffc63a3a04b30"} Jun 15 05:50:04.499262 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:04.499232 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerStarted","Data":"f480011aa025cb77d43db4be40ce8c236f37d9eb897946e81ca7413fd6104fb5"} Jun 15 05:50:05.067293 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.067258 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-796477895d-ppfwg"] Jun 15 05:50:05.069644 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.069616 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.072829 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.072790 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jun 15 05:50:05.072951 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.072924 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jun 15 05:50:05.073447 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.073401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fw4nn\"" Jun 15 05:50:05.074333 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.074296 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jun 15 05:50:05.074643 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.074625 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jun 15 05:50:05.074971 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.074949 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5s2hasi35cgj7\"" Jun 15 05:50:05.086929 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.086903 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796477895d-ppfwg"] Jun 15 05:50:05.122380 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.122347 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-secret-metrics-server-tls\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.122380 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.122392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-secret-metrics-server-client-certs\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.122606 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.122415 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-client-ca-bundle\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.122606 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.122438 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-metrics-server-audit-profiles\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.122606 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.122578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-audit-log\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.122730 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.122620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.122730 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.122658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnwt\" (UniqueName: \"kubernetes.io/projected/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-kube-api-access-ggnwt\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.223728 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.223688 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.223919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.223738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnwt\" (UniqueName: \"kubernetes.io/projected/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-kube-api-access-ggnwt\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.223919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.223813 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-secret-metrics-server-tls\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.223919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.223845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-secret-metrics-server-client-certs\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.223919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.223880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-client-ca-bundle\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.223919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.223906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-metrics-server-audit-profiles\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.224204 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.223944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-audit-log\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.224614 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.224563 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.224614 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.224582 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-audit-log\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.225244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.225193 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-metrics-server-audit-profiles\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.226935 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.226911 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-client-ca-bundle\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.226935 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.226927 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-secret-metrics-server-client-certs\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.227090 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.226964 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-secret-metrics-server-tls\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.237488 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.237460 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnwt\" (UniqueName: \"kubernetes.io/projected/9ab23e3e-6a4e-4a7a-bc5c-618bb344a263-kube-api-access-ggnwt\") pod \"metrics-server-796477895d-ppfwg\" (UID: \"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263\") " pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.362994 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.362961 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:05.385802 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.385777 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:05.505649 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.505612 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" event={"ID":"d190abf6-db4f-42e4-a063-87d280b4e39e","Type":"ContainerStarted","Data":"468aad92426baa566126b371918a003720e8691e4791e1dd991adf90bc032ccd"} Jun 15 05:50:05.506077 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.505674 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" event={"ID":"d190abf6-db4f-42e4-a063-87d280b4e39e","Type":"ContainerStarted","Data":"b75ff20404ec99e9986c2063e8fdba601441e2efd5de9daba006bd4fa3a17732"} Jun 15 05:50:05.507521 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.507497 2569 generic.go:356] "Generic (PLEG): container finished" podID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerID="209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2" exitCode=0 Jun 15 05:50:05.507681 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.507581 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2"} Jun 15 05:50:05.509370 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.509329 2569 generic.go:356] "Generic (PLEG): container finished" podID="fd928336-049e-4cca-88a1-bfdeb087f7f9" containerID="abcdc33bb4f558b8e65af058310624e0fab2632b4de03109fce72b058661864b" exitCode=0 Jun 15 05:50:05.509370 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.509362 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k5fq" event={"ID":"fd928336-049e-4cca-88a1-bfdeb087f7f9","Type":"ContainerDied","Data":"abcdc33bb4f558b8e65af058310624e0fab2632b4de03109fce72b058661864b"} Jun 15 05:50:05.528908 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.528879 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-796477895d-ppfwg"] Jun 15 05:50:05.529872 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.529809 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-678bc549d7-9xm6l" podStartSLOduration=2.475080062 podStartE2EDuration="5.529792079s" totalImagesPullingTime="3.054712017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:05.529017269 +0000 UTC m=+83.874220426" watchObservedRunningTime="2026-06-15 05:50:05.529792079 +0000 UTC m=+83.874995220" Jun 15 05:50:05.532279 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:50:05.532257 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab23e3e_6a4e_4a7a_bc5c_618bb344a263.slice/crio-274f63cd9d52e76305e3bcab7ccb159fd35b22dd717a13a4f8e395003888cd79 WatchSource:0}: Error finding container 274f63cd9d52e76305e3bcab7ccb159fd35b22dd717a13a4f8e395003888cd79: Status 404 returned error can't find the container with id 274f63cd9d52e76305e3bcab7ccb159fd35b22dd717a13a4f8e395003888cd79 Jun 15 05:50:05.613134 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:05.613101 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:06.516552 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.516504 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k5fq" event={"ID":"fd928336-049e-4cca-88a1-bfdeb087f7f9","Type":"ContainerStarted","Data":"5c775038f273514543b699485bd6c1cd9441c8239ac914070c77d0468e217814"} Jun 15 05:50:06.516970 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.516597 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2k5fq" event={"ID":"fd928336-049e-4cca-88a1-bfdeb087f7f9","Type":"ContainerStarted","Data":"ce757917d8d3b28c3b6a475b8e3f4a4171cc611b45173f81f17c8724f1cf94c9"} Jun 15 05:50:06.517965 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.517936 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" event={"ID":"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263","Type":"ContainerStarted","Data":"274f63cd9d52e76305e3bcab7ccb159fd35b22dd717a13a4f8e395003888cd79"} Jun 15 05:50:06.539947 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.539905 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2k5fq" podStartSLOduration=3.887936782 podStartE2EDuration="6.539889509s" totalImagesPullingTime="2.573326163s" totalInitContainerRuntime="78.626564ms" isStatefulPod=true podCreationTimestamp="2026-06-15 05:50:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:06.539526856 +0000 UTC m=+84.884729996" watchObservedRunningTime="2026-06-15 05:50:06.539889509 +0000 UTC m=+84.885092652" Jun 15 05:50:06.962477 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.962441 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:50:06.967156 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.965508 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:06.967156 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.965943 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:06.971148 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.971088 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-i8j4qmjo37vb\"" Jun 15 05:50:06.971941 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.971511 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pwhzs\"" Jun 15 05:50:06.971941 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.971588 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jun 15 05:50:06.971941 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.971732 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jun 15 05:50:06.971941 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.971817 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jun 15 05:50:06.972298 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.972281 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jun 15 05:50:06.972379 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.972363 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jun 15 05:50:06.972563 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.972518 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jun 15 05:50:06.972699 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.972666 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jun 15 05:50:06.972884 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.972778 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jun 15 05:50:06.972884 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.972846 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jun 15 05:50:06.973265 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.973248 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jun 15 05:50:06.973366 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.973277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jun 15 05:50:06.973850 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.973831 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jun 15 05:50:06.975200 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.975155 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jun 15 05:50:06.975973 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.975955 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jun 15 05:50:06.987396 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:06.987368 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:50:07.043442 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043393 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043583 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043583 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043496 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043583 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-config\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.043846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043957 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.043999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.044021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.044044 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.044069 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-722h5\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-kube-api-access-722h5\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.044128 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044580 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.044158 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.044580 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.044186 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.112962 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.112920 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:07.145277 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145238 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145454 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145294 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145454 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145342 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145454 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145370 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145454 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145454 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145417 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-722h5\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-kube-api-access-722h5\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145482 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145615 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145684 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.145744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.146075 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145743 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.146075 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.146075 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145812 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-config\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.146075 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.146075 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145889 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.146075 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145928 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.146075 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.145952 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.147390 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.147066 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.147390 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.147295 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.148668 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.148025 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.148668 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.148354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.150235 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.149439 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.150235 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.149608 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.151599 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.150338 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.151599 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.150447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.151599 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.150717 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.151599 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.150903 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.151599 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.151492 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.153928 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.153882 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.154771 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.154723 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-config-out\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.155339 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.155304 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.155339 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.155333 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.155862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.155815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-web-config\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.156156 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.156058 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.157213 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.157171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-config\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.157500 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.157458 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.159467 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.159416 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-722h5\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-kube-api-access-722h5\") pod \"prometheus-k8s-0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.200937 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.200802 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:07.352612 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.352555 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:07.529036 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.528940 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" event={"ID":"9ab23e3e-6a4e-4a7a-bc5c-618bb344a263","Type":"ContainerStarted","Data":"19ffd6af01a86cc502668d49ea9bf530fc764bc07892d664734353e2bf3153ca"} Jun 15 05:50:07.599496 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.599448 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" podStartSLOduration=1.191944829 podStartE2EDuration="2.599430359s" totalImagesPullingTime="1.40748553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:07.559870529 +0000 UTC m=+85.905073671" watchObservedRunningTime="2026-06-15 05:50:07.599430359 +0000 UTC m=+85.944633577" Jun 15 05:50:07.599743 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:07.599706 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:50:07.601976 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:50:07.601948 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967e29d7_24c8_4e81_8729_f7a075a79cd0.slice/crio-74cab3c89f5e098d0e1598129290b5ba0aa7c753f6dfe6026503af617779ee57 WatchSource:0}: Error finding container 74cab3c89f5e098d0e1598129290b5ba0aa7c753f6dfe6026503af617779ee57: Status 404 returned error can't find the container with id 74cab3c89f5e098d0e1598129290b5ba0aa7c753f6dfe6026503af617779ee57 Jun 15 05:50:08.534895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.534861 2569 generic.go:356] "Generic (PLEG): container finished" podID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" exitCode=0 Jun 15 05:50:08.535266 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.534948 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb"} Jun 15 05:50:08.535266 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.535032 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerStarted","Data":"74cab3c89f5e098d0e1598129290b5ba0aa7c753f6dfe6026503af617779ee57"} Jun 15 05:50:08.538681 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.538640 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerStarted","Data":"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282"} Jun 15 05:50:08.538801 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.538736 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerStarted","Data":"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa"} Jun 15 05:50:08.538801 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.538769 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerStarted","Data":"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4"} Jun 15 05:50:08.538801 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.538794 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerStarted","Data":"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772"} Jun 15 05:50:08.538941 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.538819 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerStarted","Data":"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48"} Jun 15 05:50:08.538941 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.538843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerStarted","Data":"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822"} Jun 15 05:50:08.593757 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:08.593697 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.953699845 podStartE2EDuration="7.593680019s" totalImagesPullingTime="3.61860746s" totalInitContainerRuntime="21.372714ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:50:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:08.591821888 +0000 UTC m=+86.937025029" watchObservedRunningTime="2026-06-15 05:50:08.593680019 +0000 UTC m=+86.938883159" Jun 15 05:50:09.980119 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:09.980089 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:10.044347 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:10.044311 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:10.145196 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:10.145160 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:10.262378 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:10.262296 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:10.634173 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:10.634133 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:11.035558 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:11.035505 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:11.089858 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:11.089815 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:11.667764 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:11.667740 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:11.896734 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:11.896646 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:12.124147 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:12.124114 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:12.211289 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:12.211182 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:12.438636 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:12.438600 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:12.558988 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:12.558953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerStarted","Data":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} Jun 15 05:50:12.559149 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:12.559021 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerStarted","Data":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} Jun 15 05:50:13.087988 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:13.087956 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:13.756985 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:13.756917 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:13.858246 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:13.858213 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:14.571956 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:14.571922 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerStarted","Data":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} Jun 15 05:50:14.572131 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:14.571986 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerStarted","Data":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} Jun 15 05:50:14.572131 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:14.572013 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerStarted","Data":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} Jun 15 05:50:14.572131 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:14.572037 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerStarted","Data":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} Jun 15 05:50:14.636352 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:14.636297 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.744937303 podStartE2EDuration="8.63627965s" totalImagesPullingTime="4.867008199s" totalInitContainerRuntime="24.334148ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:50:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:14.633170347 +0000 UTC m=+92.978373487" watchObservedRunningTime="2026-06-15 05:50:14.63627965 +0000 UTC m=+92.981482791" Jun 15 05:50:16.148578 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:16.148525 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:16.248934 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:16.248898 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:17.352924 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:17.352885 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:50:19.496051 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:19.496011 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:19.553722 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:19.553688 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:21.068221 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:21.068184 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:21.130219 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:21.130191 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:23.449462 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:23.449428 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8n9hm" Jun 15 05:50:25.386671 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:25.386632 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:25.387083 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:25.386710 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:27.107303 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:27.107267 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:27.165412 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:27.165381 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:31.071919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:31.071883 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:31.128474 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:31.128442 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:31.549307 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:31.549270 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:31.696129 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:31.696092 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:31.779132 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:31.779098 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:32.585420 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:32.585387 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:39.926490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:39.926453 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:39.990112 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:39.990078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:40.091074 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:40.091040 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:40.170641 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:40.170610 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:40.315200 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:40.315167 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:40.409222 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:40.409189 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:42.035637 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:42.035607 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:42.138499 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:42.138467 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:42.200407 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:42.200376 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:45.391347 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:45.391316 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:45.395158 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:45.395096 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-796477895d-ppfwg" Jun 15 05:50:55.084132 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:55.084088 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:50:55.146835 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:50:55.146798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:07.352848 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:07.352818 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:07.372510 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:07.372479 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:07.790745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:07.790716 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:19.931713 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:19.931674 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:20.754104 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:20.754059 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:21.108173 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.108133 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:51:21.108676 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.108440 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="alertmanager" containerID="cri-o://bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822" gracePeriod=120 Jun 15 05:51:21.108676 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.108511 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-metric" containerID="cri-o://d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa" gracePeriod=120 Jun 15 05:51:21.108676 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.108563 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy" containerID="cri-o://cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4" gracePeriod=120 Jun 15 05:51:21.108676 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.108587 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="prom-label-proxy" containerID="cri-o://80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282" gracePeriod=120 Jun 15 05:51:21.108676 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.108516 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-web" containerID="cri-o://f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772" gracePeriod=120 Jun 15 05:51:21.108940 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.108752 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="config-reloader" containerID="cri-o://cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48" gracePeriod=120 Jun 15 05:51:21.830492 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830458 2569 generic.go:356] "Generic (PLEG): container finished" podID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerID="80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282" exitCode=0 Jun 15 05:51:21.830492 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830483 2569 generic.go:356] "Generic (PLEG): container finished" podID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerID="cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4" exitCode=0 Jun 15 05:51:21.830492 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830493 2569 generic.go:356] "Generic (PLEG): container finished" podID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerID="cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48" exitCode=0 Jun 15 05:51:21.830736 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830503 2569 generic.go:356] "Generic (PLEG): container finished" podID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerID="bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822" exitCode=0 Jun 15 05:51:21.830736 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830529 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282"} Jun 15 05:51:21.830736 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830607 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4"} Jun 15 05:51:21.830736 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830635 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48"} Jun 15 05:51:21.830736 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:21.830660 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822"} Jun 15 05:51:22.349809 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.349784 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.482390 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482317 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-metrics-client-ca\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482390 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482359 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9s5h\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-kube-api-access-q9s5h\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482390 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482391 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482614 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482406 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-tls-assets\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482614 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482592 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-cluster-tls-config\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482708 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482664 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482708 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482692 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482808 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482727 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-config-out\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482808 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482753 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-config-volume\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482808 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482794 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-web\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482975 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482824 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:22.482975 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482853 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-trusted-ca-bundle\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482975 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482894 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-main-db\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.482975 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.482922 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-web-config\") pod \"5855cd37-1505-47dc-963b-a84fea2d4df1\" (UID: \"5855cd37-1505-47dc-963b-a84fea2d4df1\") " Jun 15 05:51:22.483440 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.483207 2569 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-metrics-client-ca\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.483783 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.483751 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:22.483930 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.483902 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 05:51:22.485601 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.485518 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:22.485601 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.485526 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-config-out" (OuterVolumeSpecName: "config-out") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 05:51:22.485895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.485857 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 05:51:22.485895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.485871 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-kube-api-access-q9s5h" (OuterVolumeSpecName: "kube-api-access-q9s5h") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "kube-api-access-q9s5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 05:51:22.485895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.485872 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:22.486349 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.486330 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:22.486624 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.486605 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:22.487358 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.487342 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-config-volume" (OuterVolumeSpecName: "config-volume") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:22.490297 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.490268 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:22.496756 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.496732 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-web-config" (OuterVolumeSpecName: "web-config") pod "5855cd37-1505-47dc-963b-a84fea2d4df1" (UID: "5855cd37-1505-47dc-963b-a84fea2d4df1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:22.583846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583811 2569 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-cluster-tls-config\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.583846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583840 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.583846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583850 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583860 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-config-out\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583868 2569 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-config-volume\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583876 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583887 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583895 2569 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5855cd37-1505-47dc-963b-a84fea2d4df1-alertmanager-main-db\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583903 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-web-config\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583910 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9s5h\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-kube-api-access-q9s5h\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583919 2569 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5855cd37-1505-47dc-963b-a84fea2d4df1-secret-alertmanager-main-tls\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.584063 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.583926 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5855cd37-1505-47dc-963b-a84fea2d4df1-tls-assets\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:22.837293 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.837259 2569 generic.go:356] "Generic (PLEG): container finished" podID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerID="d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa" exitCode=0 Jun 15 05:51:22.837293 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.837287 2569 generic.go:356] "Generic (PLEG): container finished" podID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerID="f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772" exitCode=0 Jun 15 05:51:22.837496 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.837337 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.837496 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.837347 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa"} Jun 15 05:51:22.837496 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.837411 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772"} Jun 15 05:51:22.837496 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.837443 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5855cd37-1505-47dc-963b-a84fea2d4df1","Type":"ContainerDied","Data":"f480011aa025cb77d43db4be40ce8c236f37d9eb897946e81ca7413fd6104fb5"} Jun 15 05:51:22.837496 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.837480 2569 scope.go:117] "RemoveContainer" containerID="80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282" Jun 15 05:51:22.847937 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.847910 2569 scope.go:117] "RemoveContainer" containerID="d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa" Jun 15 05:51:22.855195 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.855170 2569 scope.go:117] "RemoveContainer" containerID="cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4" Jun 15 05:51:22.861849 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.861832 2569 scope.go:117] "RemoveContainer" containerID="f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772" Jun 15 05:51:22.868735 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.868689 2569 scope.go:117] "RemoveContainer" containerID="cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48" Jun 15 05:51:22.869729 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.869695 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:51:22.874598 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.874571 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:51:22.886180 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.885444 2569 scope.go:117] "RemoveContainer" containerID="bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822" Jun 15 05:51:22.905023 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.904994 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:51:22.905299 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905287 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="init-config-reloader" Jun 15 05:51:22.905341 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905301 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="init-config-reloader" Jun 15 05:51:22.905341 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905309 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-metric" Jun 15 05:51:22.905341 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905315 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-metric" Jun 15 05:51:22.905341 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905323 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="prom-label-proxy" Jun 15 05:51:22.905341 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905328 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="prom-label-proxy" Jun 15 05:51:22.905341 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905336 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="alertmanager" Jun 15 05:51:22.905341 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905341 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="alertmanager" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905354 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-web" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905359 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-web" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905368 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905373 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905381 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="config-reloader" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905386 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="config-reloader" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905428 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-metric" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905435 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="prom-label-proxy" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905442 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="alertmanager" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905448 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="config-reloader" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905455 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy-web" Jun 15 05:51:22.905565 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905461 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" containerName="kube-rbac-proxy" Jun 15 05:51:22.905949 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.905765 2569 scope.go:117] "RemoveContainer" containerID="209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2" Jun 15 05:51:22.910349 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.910327 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.913767 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.913745 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jun 15 05:51:22.913985 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.913800 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tb5vz\"" Jun 15 05:51:22.914082 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914001 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jun 15 05:51:22.914192 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914111 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jun 15 05:51:22.914295 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jun 15 05:51:22.914355 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jun 15 05:51:22.914536 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914314 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jun 15 05:51:22.914536 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914476 2569 scope.go:117] "RemoveContainer" containerID="80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282" Jun 15 05:51:22.914719 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914695 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jun 15 05:51:22.914834 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914700 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jun 15 05:51:22.914892 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:22.914819 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282\": container with ID starting with 80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282 not found: ID does not exist" containerID="80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282" Jun 15 05:51:22.914953 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914886 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282"} err="failed to get container status \"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282\": rpc error: code = NotFound desc = could not find container \"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282\": container with ID starting with 80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282 not found: ID does not exist" Jun 15 05:51:22.914953 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.914929 2569 scope.go:117] "RemoveContainer" containerID="d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa" Jun 15 05:51:22.916960 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:22.916935 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa\": container with ID starting with d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa not found: ID does not exist" containerID="d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa" Jun 15 05:51:22.917062 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.916967 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa"} err="failed to get container status \"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa\": rpc error: code = NotFound desc = could not find container \"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa\": container with ID starting with d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa not found: ID does not exist" Jun 15 05:51:22.917062 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.916993 2569 scope.go:117] "RemoveContainer" containerID="cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4" Jun 15 05:51:22.917341 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:22.917315 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4\": container with ID starting with cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4 not found: ID does not exist" containerID="cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4" Jun 15 05:51:22.917440 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.917357 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4"} err="failed to get container status \"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4\": rpc error: code = NotFound desc = could not find container \"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4\": container with ID starting with cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4 not found: ID does not exist" Jun 15 05:51:22.917440 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.917382 2569 scope.go:117] "RemoveContainer" containerID="f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772" Jun 15 05:51:22.917721 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:22.917694 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772\": container with ID starting with f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772 not found: ID does not exist" containerID="f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772" Jun 15 05:51:22.917798 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.917726 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772"} err="failed to get container status \"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772\": rpc error: code = NotFound desc = could not find container \"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772\": container with ID starting with f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772 not found: ID does not exist" Jun 15 05:51:22.917798 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.917748 2569 scope.go:117] "RemoveContainer" containerID="cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48" Jun 15 05:51:22.918013 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:22.917995 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48\": container with ID starting with cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48 not found: ID does not exist" containerID="cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48" Jun 15 05:51:22.918079 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918017 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48"} err="failed to get container status \"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48\": rpc error: code = NotFound desc = could not find container \"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48\": container with ID starting with cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48 not found: ID does not exist" Jun 15 05:51:22.918079 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918030 2569 scope.go:117] "RemoveContainer" containerID="bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822" Jun 15 05:51:22.918264 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:22.918245 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822\": container with ID starting with bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822 not found: ID does not exist" containerID="bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822" Jun 15 05:51:22.918360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918267 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822"} err="failed to get container status \"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822\": rpc error: code = NotFound desc = could not find container \"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822\": container with ID starting with bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822 not found: ID does not exist" Jun 15 05:51:22.918360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918286 2569 scope.go:117] "RemoveContainer" containerID="209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2" Jun 15 05:51:22.918623 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:22.918600 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2\": container with ID starting with 209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2 not found: ID does not exist" containerID="209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2" Jun 15 05:51:22.918680 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918629 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2"} err="failed to get container status \"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2\": rpc error: code = NotFound desc = could not find container \"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2\": container with ID starting with 209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2 not found: ID does not exist" Jun 15 05:51:22.918680 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918645 2569 scope.go:117] "RemoveContainer" containerID="80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282" Jun 15 05:51:22.918935 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918910 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282"} err="failed to get container status \"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282\": rpc error: code = NotFound desc = could not find container \"80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282\": container with ID starting with 80d36823b26184661c4c0e060ff1aa544567e3fae7f8eb1ddb74e1b3ba3bc282 not found: ID does not exist" Jun 15 05:51:22.919022 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.918937 2569 scope.go:117] "RemoveContainer" containerID="d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa" Jun 15 05:51:22.919245 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.919224 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa"} err="failed to get container status \"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa\": rpc error: code = NotFound desc = could not find container \"d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa\": container with ID starting with d436c07bbbf22283610a047e461344c6e0dd7979c49c6ef3269630418e2159fa not found: ID does not exist" Jun 15 05:51:22.919313 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.919246 2569 scope.go:117] "RemoveContainer" containerID="cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4" Jun 15 05:51:22.919497 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.919476 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4"} err="failed to get container status \"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4\": rpc error: code = NotFound desc = could not find container \"cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4\": container with ID starting with cc313f96a512404289feaa741e9bf49370d193e42e54707a79d00334020264f4 not found: ID does not exist" Jun 15 05:51:22.919497 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.919495 2569 scope.go:117] "RemoveContainer" containerID="f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772" Jun 15 05:51:22.919767 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.919742 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772"} err="failed to get container status \"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772\": rpc error: code = NotFound desc = could not find container \"f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772\": container with ID starting with f5c983b2c867ccd4530912d1272cface6e6620877393f3f49fbe64da330f9772 not found: ID does not exist" Jun 15 05:51:22.919767 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.919766 2569 scope.go:117] "RemoveContainer" containerID="cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48" Jun 15 05:51:22.920122 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.920010 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48"} err="failed to get container status \"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48\": rpc error: code = NotFound desc = could not find container \"cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48\": container with ID starting with cfe70bbd26dd13eff1e2d1deb3c71a60b46470b26855f5eab4d5f75428baca48 not found: ID does not exist" Jun 15 05:51:22.920122 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.920041 2569 scope.go:117] "RemoveContainer" containerID="bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822" Jun 15 05:51:22.920609 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.920584 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822"} err="failed to get container status \"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822\": rpc error: code = NotFound desc = could not find container \"bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822\": container with ID starting with bb47a1408c8786e90421f7f8ae9d2e34d094aa2e9eedd328dd572c3188c07822 not found: ID does not exist" Jun 15 05:51:22.920708 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.920609 2569 scope.go:117] "RemoveContainer" containerID="209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2" Jun 15 05:51:22.920909 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.920881 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2"} err="failed to get container status \"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2\": rpc error: code = NotFound desc = could not find container \"209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2\": container with ID starting with 209449d384e559585c5ac55d76b4d07519f060427b4ac16ff83a56a3fbc775a2 not found: ID does not exist" Jun 15 05:51:22.921367 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.921352 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jun 15 05:51:22.926161 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.926141 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:51:22.986671 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986693 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhjj9\" (UniqueName: \"kubernetes.io/projected/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-kube-api-access-xhjj9\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986740 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-config-volume\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-config-out\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986837 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.986860 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986851 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.987230 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986922 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.987230 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-web-config\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.987230 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.986997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:22.987230 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:22.987041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088101 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.087998 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-web-config\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088101 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088039 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088101 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088193 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhjj9\" (UniqueName: \"kubernetes.io/projected/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-kube-api-access-xhjj9\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-config-volume\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088383 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088261 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088701 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-config-out\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088701 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088701 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088648 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088701 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088671 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.088915 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.088715 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.089440 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.089113 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.089578 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.089537 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.089959 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.089913 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.091179 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.091152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.091774 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.091649 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.091774 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.091693 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.091936 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.091830 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-web-config\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.092001 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.091951 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.092055 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.092029 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-config-volume\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.092130 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.092114 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.092130 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.092122 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-config-out\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.092714 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.092700 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.097264 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.097246 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhjj9\" (UniqueName: \"kubernetes.io/projected/9bea2f72-a04f-484d-97b5-6d3d2bee1c64-kube-api-access-xhjj9\") pod \"alertmanager-main-0\" (UID: \"9bea2f72-a04f-484d-97b5-6d3d2bee1c64\") " pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.223736 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.223700 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jun 15 05:51:23.361591 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.361487 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jun 15 05:51:23.364422 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:51:23.364395 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bea2f72_a04f_484d_97b5_6d3d2bee1c64.slice/crio-5f2962751b7813b35133e82ad924be94eb1ea463f21f86c2710c9066482bb44f WatchSource:0}: Error finding container 5f2962751b7813b35133e82ad924be94eb1ea463f21f86c2710c9066482bb44f: Status 404 returned error can't find the container with id 5f2962751b7813b35133e82ad924be94eb1ea463f21f86c2710c9066482bb44f Jun 15 05:51:23.842628 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.842599 2569 generic.go:356] "Generic (PLEG): container finished" podID="9bea2f72-a04f-484d-97b5-6d3d2bee1c64" containerID="70901e4cb0ce3f12070c0053730d9925a8c4f49709eb31344c69da980e6d4e10" exitCode=0 Jun 15 05:51:23.842781 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.842657 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerDied","Data":"70901e4cb0ce3f12070c0053730d9925a8c4f49709eb31344c69da980e6d4e10"} Jun 15 05:51:23.842781 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:23.842721 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerStarted","Data":"5f2962751b7813b35133e82ad924be94eb1ea463f21f86c2710c9066482bb44f"} Jun 15 05:51:24.056710 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.056683 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5855cd37-1505-47dc-963b-a84fea2d4df1" path="/var/lib/kubelet/pods/5855cd37-1505-47dc-963b-a84fea2d4df1/volumes" Jun 15 05:51:24.850532 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.850496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerStarted","Data":"0b7eeba3a4b69be6e5c9b574424fb046ef2479f58f0cae42795cf1ef6c91bdf1"} Jun 15 05:51:24.850923 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.850601 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerStarted","Data":"b9c217a0961113640cec1e97ce98db39dd2c3d800f23c4e0efa2bc295b11a46e"} Jun 15 05:51:24.850923 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.850627 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerStarted","Data":"2c15bce5c5e853dfdada536c87c81778bd0c474cc9769c7d70a57bcdc4e828cf"} Jun 15 05:51:24.850923 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.850651 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerStarted","Data":"673c909e90a2f41d977f39cdf61b290eb843d7f0b1f8c7041c674a95ef97bc24"} Jun 15 05:51:24.850923 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.850673 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerStarted","Data":"666eb3624655e2436f607fde68ed637cd51e9eb6b0b685a8265f03886989979f"} Jun 15 05:51:24.850923 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.850695 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9bea2f72-a04f-484d-97b5-6d3d2bee1c64","Type":"ContainerStarted","Data":"606c2339f181ea152a2603ee7b32e85259f8fd72bd307b1ddbcafc3f104fe868"} Jun 15 05:51:24.884649 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:24.884594 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.869667154 podStartE2EDuration="2.884566046s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.898892ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:51:24.880620572 +0000 UTC m=+163.225823713" watchObservedRunningTime="2026-06-15 05:51:24.884566046 +0000 UTC m=+163.229769191" Jun 15 05:51:25.347112 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.347077 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:51:25.347642 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.347370 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="prometheus" containerID="cri-o://26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" gracePeriod=600 Jun 15 05:51:25.347642 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.347413 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="thanos-sidecar" containerID="cri-o://1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" gracePeriod=600 Jun 15 05:51:25.347642 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.347423 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" gracePeriod=600 Jun 15 05:51:25.347642 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.347376 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy" containerID="cri-o://cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" gracePeriod=600 Jun 15 05:51:25.347642 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.347462 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="config-reloader" containerID="cri-o://043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" gracePeriod=600 Jun 15 05:51:25.347642 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.347429 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-web" containerID="cri-o://1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" gracePeriod=600 Jun 15 05:51:25.607811 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.607783 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:25.713605 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713576 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-metrics-client-certs\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713762 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713612 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713762 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713661 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-1\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713864 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713768 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-thanos-prometheus-http-client-file\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713864 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713844 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-kube-rbac-proxy\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713981 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713872 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-722h5\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-kube-api-access-722h5\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713981 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713908 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-config-out\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713981 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713942 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-db\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.713981 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713947 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:25.714164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.713983 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-tls-assets\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714007 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-grpc-tls\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714036 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-kubelet-serving-ca-bundle\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714062 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-2\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714091 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-tls\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714122 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-serving-certs-ca-bundle\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714164 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714147 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-web-config\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714474 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714183 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-metrics-client-ca\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714474 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714211 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-trusted-ca-bundle\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714474 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714244 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714474 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714272 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-0\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714474 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714303 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-config\") pod \"967e29d7-24c8-4e81-8729-f7a075a79cd0\" (UID: \"967e29d7-24c8-4e81-8729-f7a075a79cd0\") " Jun 15 05:51:25.714737 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.714581 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.715382 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.716586 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.716741 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.716734 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.716805 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.717078 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-kube-api-access-722h5" (OuterVolumeSpecName: "kube-api-access-722h5") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "kube-api-access-722h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.717123 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.717220 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-config-out" (OuterVolumeSpecName: "config-out") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.717293 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.717401 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.718682 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:25.718803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.718758 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.719847 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.719808 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.720242 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.720209 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.720469 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.720439 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.720888 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.720867 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-config" (OuterVolumeSpecName: "config") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.721168 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.721146 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 05:51:25.721363 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.721340 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:51:25.731269 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.731249 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-web-config" (OuterVolumeSpecName: "web-config") pod "967e29d7-24c8-4e81-8729-f7a075a79cd0" (UID: "967e29d7-24c8-4e81-8729-f7a075a79cd0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 05:51:25.815121 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815084 2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-kube-rbac-proxy\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815121 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815114 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-722h5\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-kube-api-access-722h5\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815121 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815124 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-config-out\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815121 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815133 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-db\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815143 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/967e29d7-24c8-4e81-8729-f7a075a79cd0-tls-assets\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815151 2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-grpc-tls\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815161 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815170 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815178 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815186 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815194 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-web-config\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815202 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-configmap-metrics-client-ca\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815212 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815220 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815229 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/967e29d7-24c8-4e81-8729-f7a075a79cd0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815238 2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-config\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815246 2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-metrics-client-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815255 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.815382 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.815264 2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/967e29d7-24c8-4e81-8729-f7a075a79cd0-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:51:25.858049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.857967 2569 generic.go:356] "Generic (PLEG): container finished" podID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" exitCode=0 Jun 15 05:51:25.858049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.857988 2569 generic.go:356] "Generic (PLEG): container finished" podID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" exitCode=0 Jun 15 05:51:25.858049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.857997 2569 generic.go:356] "Generic (PLEG): container finished" podID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" exitCode=0 Jun 15 05:51:25.858049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858003 2569 generic.go:356] "Generic (PLEG): container finished" podID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" exitCode=0 Jun 15 05:51:25.858049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858009 2569 generic.go:356] "Generic (PLEG): container finished" podID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" exitCode=0 Jun 15 05:51:25.858049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858014 2569 generic.go:356] "Generic (PLEG): container finished" podID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" exitCode=0 Jun 15 05:51:25.858049 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858047 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858116 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858144 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858168 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858189 2569 scope.go:117] "RemoveContainer" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858326 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} Jun 15 05:51:25.858704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.858367 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"967e29d7-24c8-4e81-8729-f7a075a79cd0","Type":"ContainerDied","Data":"74cab3c89f5e098d0e1598129290b5ba0aa7c753f6dfe6026503af617779ee57"} Jun 15 05:51:25.866360 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.866341 2569 scope.go:117] "RemoveContainer" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.873270 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.873253 2569 scope.go:117] "RemoveContainer" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.880094 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.880075 2569 scope.go:117] "RemoveContainer" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.883532 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.883504 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:51:25.890888 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.890867 2569 scope.go:117] "RemoveContainer" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.891122 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.891095 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:51:25.897772 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.897752 2569 scope.go:117] "RemoveContainer" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.905066 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.905044 2569 scope.go:117] "RemoveContainer" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.911824 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.911806 2569 scope.go:117] "RemoveContainer" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.912066 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:25.912048 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": container with ID starting with d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506 not found: ID does not exist" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.912140 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912076 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} err="failed to get container status \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": rpc error: code = NotFound desc = could not find container \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": container with ID starting with d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506 not found: ID does not exist" Jun 15 05:51:25.912140 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912104 2569 scope.go:117] "RemoveContainer" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.912361 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:25.912344 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": container with ID starting with cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705 not found: ID does not exist" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.912396 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912367 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} err="failed to get container status \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": rpc error: code = NotFound desc = could not find container \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": container with ID starting with cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705 not found: ID does not exist" Jun 15 05:51:25.912396 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912384 2569 scope.go:117] "RemoveContainer" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.912617 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:25.912600 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": container with ID starting with 1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168 not found: ID does not exist" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.912690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912627 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} err="failed to get container status \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": rpc error: code = NotFound desc = could not find container \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": container with ID starting with 1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168 not found: ID does not exist" Jun 15 05:51:25.912690 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912641 2569 scope.go:117] "RemoveContainer" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.912920 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:25.912898 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": container with ID starting with 1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4 not found: ID does not exist" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.912981 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912925 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} err="failed to get container status \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": rpc error: code = NotFound desc = could not find container \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": container with ID starting with 1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4 not found: ID does not exist" Jun 15 05:51:25.912981 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.912948 2569 scope.go:117] "RemoveContainer" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.913155 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:25.913139 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": container with ID starting with 043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563 not found: ID does not exist" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.913198 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913159 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} err="failed to get container status \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": rpc error: code = NotFound desc = could not find container \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": container with ID starting with 043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563 not found: ID does not exist" Jun 15 05:51:25.913198 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913172 2569 scope.go:117] "RemoveContainer" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.913376 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:25.913359 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": container with ID starting with 26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663 not found: ID does not exist" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.913415 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913383 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} err="failed to get container status \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": rpc error: code = NotFound desc = could not find container \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": container with ID starting with 26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663 not found: ID does not exist" Jun 15 05:51:25.913415 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913397 2569 scope.go:117] "RemoveContainer" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.913685 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:51:25.913670 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": container with ID starting with 06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb not found: ID does not exist" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.913731 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913686 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb"} err="failed to get container status \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": rpc error: code = NotFound desc = could not find container \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": container with ID starting with 06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb not found: ID does not exist" Jun 15 05:51:25.913731 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913698 2569 scope.go:117] "RemoveContainer" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.913935 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913915 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} err="failed to get container status \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": rpc error: code = NotFound desc = could not find container \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": container with ID starting with d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506 not found: ID does not exist" Jun 15 05:51:25.913986 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.913943 2569 scope.go:117] "RemoveContainer" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.914170 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914153 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} err="failed to get container status \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": rpc error: code = NotFound desc = could not find container \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": container with ID starting with cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705 not found: ID does not exist" Jun 15 05:51:25.914208 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914171 2569 scope.go:117] "RemoveContainer" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.914422 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914385 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} err="failed to get container status \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": rpc error: code = NotFound desc = could not find container \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": container with ID starting with 1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168 not found: ID does not exist" Jun 15 05:51:25.914472 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914421 2569 scope.go:117] "RemoveContainer" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.914622 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914603 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} err="failed to get container status \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": rpc error: code = NotFound desc = could not find container \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": container with ID starting with 1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4 not found: ID does not exist" Jun 15 05:51:25.914728 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914621 2569 scope.go:117] "RemoveContainer" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.914850 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914832 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} err="failed to get container status \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": rpc error: code = NotFound desc = could not find container \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": container with ID starting with 043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563 not found: ID does not exist" Jun 15 05:51:25.914899 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.914850 2569 scope.go:117] "RemoveContainer" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.915040 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915020 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} err="failed to get container status \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": rpc error: code = NotFound desc = could not find container \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": container with ID starting with 26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663 not found: ID does not exist" Jun 15 05:51:25.915085 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915041 2569 scope.go:117] "RemoveContainer" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.915224 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915209 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb"} err="failed to get container status \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": rpc error: code = NotFound desc = could not find container \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": container with ID starting with 06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb not found: ID does not exist" Jun 15 05:51:25.915270 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915225 2569 scope.go:117] "RemoveContainer" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.915427 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915411 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} err="failed to get container status \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": rpc error: code = NotFound desc = could not find container \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": container with ID starting with d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506 not found: ID does not exist" Jun 15 05:51:25.915465 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915427 2569 scope.go:117] "RemoveContainer" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.915641 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915623 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} err="failed to get container status \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": rpc error: code = NotFound desc = could not find container \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": container with ID starting with cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705 not found: ID does not exist" Jun 15 05:51:25.915704 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915641 2569 scope.go:117] "RemoveContainer" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.915865 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915847 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} err="failed to get container status \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": rpc error: code = NotFound desc = could not find container \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": container with ID starting with 1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168 not found: ID does not exist" Jun 15 05:51:25.915865 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.915865 2569 scope.go:117] "RemoveContainer" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.916069 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916052 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} err="failed to get container status \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": rpc error: code = NotFound desc = could not find container \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": container with ID starting with 1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4 not found: ID does not exist" Jun 15 05:51:25.916069 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916068 2569 scope.go:117] "RemoveContainer" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.916262 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916246 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} err="failed to get container status \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": rpc error: code = NotFound desc = could not find container \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": container with ID starting with 043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563 not found: ID does not exist" Jun 15 05:51:25.916262 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916261 2569 scope.go:117] "RemoveContainer" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.916468 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916451 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} err="failed to get container status \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": rpc error: code = NotFound desc = could not find container \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": container with ID starting with 26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663 not found: ID does not exist" Jun 15 05:51:25.916518 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916469 2569 scope.go:117] "RemoveContainer" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.916705 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916684 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb"} err="failed to get container status \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": rpc error: code = NotFound desc = could not find container \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": container with ID starting with 06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb not found: ID does not exist" Jun 15 05:51:25.916757 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916705 2569 scope.go:117] "RemoveContainer" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.916913 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916896 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} err="failed to get container status \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": rpc error: code = NotFound desc = could not find container \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": container with ID starting with d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506 not found: ID does not exist" Jun 15 05:51:25.916968 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.916912 2569 scope.go:117] "RemoveContainer" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.917143 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917123 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} err="failed to get container status \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": rpc error: code = NotFound desc = could not find container \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": container with ID starting with cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705 not found: ID does not exist" Jun 15 05:51:25.917143 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917142 2569 scope.go:117] "RemoveContainer" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.917344 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917328 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} err="failed to get container status \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": rpc error: code = NotFound desc = could not find container \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": container with ID starting with 1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168 not found: ID does not exist" Jun 15 05:51:25.917385 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917344 2569 scope.go:117] "RemoveContainer" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.917562 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917528 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} err="failed to get container status \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": rpc error: code = NotFound desc = could not find container \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": container with ID starting with 1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4 not found: ID does not exist" Jun 15 05:51:25.917624 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917561 2569 scope.go:117] "RemoveContainer" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.917792 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917775 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} err="failed to get container status \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": rpc error: code = NotFound desc = could not find container \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": container with ID starting with 043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563 not found: ID does not exist" Jun 15 05:51:25.917846 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917792 2569 scope.go:117] "RemoveContainer" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.917964 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917950 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} err="failed to get container status \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": rpc error: code = NotFound desc = could not find container \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": container with ID starting with 26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663 not found: ID does not exist" Jun 15 05:51:25.918001 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.917966 2569 scope.go:117] "RemoveContainer" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.918165 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918149 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb"} err="failed to get container status \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": rpc error: code = NotFound desc = could not find container \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": container with ID starting with 06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb not found: ID does not exist" Jun 15 05:51:25.918210 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918165 2569 scope.go:117] "RemoveContainer" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.918381 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918365 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} err="failed to get container status \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": rpc error: code = NotFound desc = could not find container \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": container with ID starting with d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506 not found: ID does not exist" Jun 15 05:51:25.918428 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918382 2569 scope.go:117] "RemoveContainer" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.918605 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918587 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} err="failed to get container status \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": rpc error: code = NotFound desc = could not find container \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": container with ID starting with cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705 not found: ID does not exist" Jun 15 05:51:25.918655 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918606 2569 scope.go:117] "RemoveContainer" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.918838 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918823 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} err="failed to get container status \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": rpc error: code = NotFound desc = could not find container \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": container with ID starting with 1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168 not found: ID does not exist" Jun 15 05:51:25.918881 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.918838 2569 scope.go:117] "RemoveContainer" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.919042 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919021 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} err="failed to get container status \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": rpc error: code = NotFound desc = could not find container \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": container with ID starting with 1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4 not found: ID does not exist" Jun 15 05:51:25.919042 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919042 2569 scope.go:117] "RemoveContainer" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.919243 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919225 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} err="failed to get container status \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": rpc error: code = NotFound desc = could not find container \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": container with ID starting with 043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563 not found: ID does not exist" Jun 15 05:51:25.919304 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919242 2569 scope.go:117] "RemoveContainer" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.919458 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919441 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} err="failed to get container status \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": rpc error: code = NotFound desc = could not find container \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": container with ID starting with 26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663 not found: ID does not exist" Jun 15 05:51:25.919458 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919457 2569 scope.go:117] "RemoveContainer" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.919670 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919654 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb"} err="failed to get container status \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": rpc error: code = NotFound desc = could not find container \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": container with ID starting with 06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb not found: ID does not exist" Jun 15 05:51:25.919670 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919669 2569 scope.go:117] "RemoveContainer" containerID="d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506" Jun 15 05:51:25.919853 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919835 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506"} err="failed to get container status \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": rpc error: code = NotFound desc = could not find container \"d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506\": container with ID starting with d990633a372c652715cf80b407e936ed76b30a21a6e6b6a804a12d21e607f506 not found: ID does not exist" Jun 15 05:51:25.919895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.919854 2569 scope.go:117] "RemoveContainer" containerID="cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705" Jun 15 05:51:25.920048 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920031 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705"} err="failed to get container status \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": rpc error: code = NotFound desc = could not find container \"cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705\": container with ID starting with cc62774c4a54277e07eb6a0dd35feec0c29c5607a71eaab307320004c103d705 not found: ID does not exist" Jun 15 05:51:25.920093 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920048 2569 scope.go:117] "RemoveContainer" containerID="1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168" Jun 15 05:51:25.920245 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920227 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168"} err="failed to get container status \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": rpc error: code = NotFound desc = could not find container \"1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168\": container with ID starting with 1e41f5c2712c1f2690621bb5102d3a8826c0ce653611db69d44b47ce53985168 not found: ID does not exist" Jun 15 05:51:25.920283 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920247 2569 scope.go:117] "RemoveContainer" containerID="1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4" Jun 15 05:51:25.920434 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920417 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4"} err="failed to get container status \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": rpc error: code = NotFound desc = could not find container \"1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4\": container with ID starting with 1bfe6e6eff8b5f8f61e702976da18189226febd203cfa7ec8a0a7892dd1b16c4 not found: ID does not exist" Jun 15 05:51:25.920434 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920432 2569 scope.go:117] "RemoveContainer" containerID="043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563" Jun 15 05:51:25.920654 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920637 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563"} err="failed to get container status \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": rpc error: code = NotFound desc = could not find container \"043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563\": container with ID starting with 043657ad19d7c04f605c174a3e19b030ce24b0200c541bce9e0de5eb1a646563 not found: ID does not exist" Jun 15 05:51:25.920654 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920654 2569 scope.go:117] "RemoveContainer" containerID="26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663" Jun 15 05:51:25.920906 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920890 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663"} err="failed to get container status \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": rpc error: code = NotFound desc = could not find container \"26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663\": container with ID starting with 26d4692077a27e22b0ee71edd7bf69d91ae8a78ab4a132aee6d97172b3eda663 not found: ID does not exist" Jun 15 05:51:25.920961 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.920909 2569 scope.go:117] "RemoveContainer" containerID="06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb" Jun 15 05:51:25.921156 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.921135 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb"} err="failed to get container status \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": rpc error: code = NotFound desc = could not find container \"06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb\": container with ID starting with 06b1d6405b9d93e80d25f71330a896438eb3462c179a8140faa06b0bc20694eb not found: ID does not exist" Jun 15 05:51:25.922891 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.922870 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:51:25.923245 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923227 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="prometheus" Jun 15 05:51:25.923245 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923245 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="prometheus" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923261 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="init-config-reloader" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923268 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="init-config-reloader" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923278 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="config-reloader" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923286 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="config-reloader" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923295 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-web" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923303 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-web" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923312 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="thanos-sidecar" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923317 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="thanos-sidecar" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923323 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923328 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923334 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-thanos" Jun 15 05:51:25.923364 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923339 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-thanos" Jun 15 05:51:25.923703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923384 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy" Jun 15 05:51:25.923703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923395 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="config-reloader" Jun 15 05:51:25.923703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923400 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-web" Jun 15 05:51:25.923703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923405 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="thanos-sidecar" Jun 15 05:51:25.923703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923410 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="prometheus" Jun 15 05:51:25.923703 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.923415 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" containerName="kube-rbac-proxy-thanos" Jun 15 05:51:25.928498 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.928480 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:25.932176 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.931636 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jun 15 05:51:25.932176 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.931969 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jun 15 05:51:25.932176 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932153 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932419 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932426 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932486 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932657 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932678 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-i8j4qmjo37vb\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932732 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932734 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jun 15 05:51:25.932806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932740 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jun 15 05:51:25.933375 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932906 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jun 15 05:51:25.933375 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.932933 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pwhzs\"" Jun 15 05:51:25.933375 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.933371 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jun 15 05:51:25.940350 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.940330 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jun 15 05:51:25.942359 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.942341 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jun 15 05:51:25.952946 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:25.952925 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:51:26.016598 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016561 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.016765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016708 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.016765 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016738 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.016919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016767 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.016919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016806 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.016919 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016917 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016959 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016978 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.016997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81f4c287-57a0-4cce-bd05-db5ea473b276-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017081 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017376 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017097 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-web-config\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017376 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017376 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6r8\" (UniqueName: \"kubernetes.io/projected/81f4c287-57a0-4cce-bd05-db5ea473b276-kube-api-access-fw6r8\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017376 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017176 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81f4c287-57a0-4cce-bd05-db5ea473b276-config-out\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017376 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017715 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017474 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-config\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017715 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017519 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.017715 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.017564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.055205 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.055177 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="967e29d7-24c8-4e81-8729-f7a075a79cd0" path="/var/lib/kubelet/pods/967e29d7-24c8-4e81-8729-f7a075a79cd0/volumes" Jun 15 05:51:26.118315 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118315 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118315 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81f4c287-57a0-4cce-bd05-db5ea473b276-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118594 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118594 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118357 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118594 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118388 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-web-config\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118594 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118419 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118594 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6r8\" (UniqueName: \"kubernetes.io/projected/81f4c287-57a0-4cce-bd05-db5ea473b276-kube-api-access-fw6r8\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.118594 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.118468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81f4c287-57a0-4cce-bd05-db5ea473b276-config-out\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.119058 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119031 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119099 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-config\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119281 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119283 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119335 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119401 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119419 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.119686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.120482 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.121128 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.121222 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.121893 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.121689 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.122806 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.121898 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-config\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.122873 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.122836 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-web-config\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.122985 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.122941 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/81f4c287-57a0-4cce-bd05-db5ea473b276-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.122985 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.122958 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.123123 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.123019 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/81f4c287-57a0-4cce-bd05-db5ea473b276-config-out\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.123123 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.123055 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.123971 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.123929 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.124118 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.124098 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.124189 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.124159 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.124381 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.124364 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.124604 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.124587 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/81f4c287-57a0-4cce-bd05-db5ea473b276-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.124604 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.124596 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/81f4c287-57a0-4cce-bd05-db5ea473b276-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.130196 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.130179 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6r8\" (UniqueName: \"kubernetes.io/projected/81f4c287-57a0-4cce-bd05-db5ea473b276-kube-api-access-fw6r8\") pod \"prometheus-k8s-0\" (UID: \"81f4c287-57a0-4cce-bd05-db5ea473b276\") " pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.244592 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.244555 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:26.386588 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.386562 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jun 15 05:51:26.388424 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:51:26.388386 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f4c287_57a0_4cce_bd05_db5ea473b276.slice/crio-3e07e8355a28cad7c367d679974aba42c8cfc870c3d0013a5fa4d095065db2b5 WatchSource:0}: Error finding container 3e07e8355a28cad7c367d679974aba42c8cfc870c3d0013a5fa4d095065db2b5: Status 404 returned error can't find the container with id 3e07e8355a28cad7c367d679974aba42c8cfc870c3d0013a5fa4d095065db2b5 Jun 15 05:51:26.864775 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.864737 2569 generic.go:356] "Generic (PLEG): container finished" podID="81f4c287-57a0-4cce-bd05-db5ea473b276" containerID="96d3f9045e84336cd051d1a54e82b743ee8496ae448133c84648a98193229586" exitCode=0 Jun 15 05:51:26.864775 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.864778 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerDied","Data":"96d3f9045e84336cd051d1a54e82b743ee8496ae448133c84648a98193229586"} Jun 15 05:51:26.865318 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:26.864850 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerStarted","Data":"3e07e8355a28cad7c367d679974aba42c8cfc870c3d0013a5fa4d095065db2b5"} Jun 15 05:51:27.872426 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:27.872384 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerStarted","Data":"96ca3c265012dc35baa107db48e254cabceba5e3f34f4b58c23c835c8fbe4b20"} Jun 15 05:51:27.872862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:27.872503 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerStarted","Data":"06e87bb8a84a2e25054610d26785b2724fba027afd667601153eef2f4f438466"} Jun 15 05:51:27.872862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:27.872567 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerStarted","Data":"88b14edb30b046d3906d703317ca5a006ccc384ff2afa79aad8b5acacf8186bd"} Jun 15 05:51:27.872862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:27.872605 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerStarted","Data":"238eba4d56307be12c13d8058888d748b89bb1c7cebd1365fb790ce3c6a3c43b"} Jun 15 05:51:27.872862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:27.872639 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerStarted","Data":"977772d2cc2fc53f20044755b4e4be2610292b4276e68c8751a661067621b48a"} Jun 15 05:51:27.872862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:27.872676 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"81f4c287-57a0-4cce-bd05-db5ea473b276","Type":"ContainerStarted","Data":"9cef8c3cd95dbb75e958ff1c63317e1e4ee0dc0b5b86d6e5702a6808a416c19d"} Jun 15 05:51:27.906685 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:27.906619 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.883869457 podStartE2EDuration="2.906597012s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.727555ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:51:27.905926136 +0000 UTC m=+166.251129319" watchObservedRunningTime="2026-06-15 05:51:27.906597012 +0000 UTC m=+166.251800155" Jun 15 05:51:30.503715 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:30.503680 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:30.557143 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:30.557103 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:31.245520 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:31.245435 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:51:41.254699 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:41.254666 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:41.355499 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:41.355464 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:41.412517 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:41.412484 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:51:57.814586 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:51:57.814527 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:12.943574 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:12.943521 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:13.002490 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:13.002462 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:26.245071 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:26.245033 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:52:26.260587 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:26.260529 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:52:27.103179 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:27.103150 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jun 15 05:52:35.253796 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:35.253764 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:35.355944 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:35.355909 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:47.180157 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:47.180121 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:47.234751 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:47.234719 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:55.080610 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:55.080578 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:55.180859 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:55.180823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:55.399599 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:55.399517 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:57.008836 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:57.008810 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:57.108136 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:57.108109 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:52:57.204484 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:52:57.204453 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:10.236683 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:10.236605 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:10.295219 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:10.295190 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:21.565267 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:21.565231 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:21.620927 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:21.620891 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:21.688273 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:21.688242 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:38.354908 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:38.354869 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:38.409833 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:38.409797 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:38.729470 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:38.729397 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:39.035228 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.035199 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484"] Jun 15 05:53:39.038502 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.038484 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.040867 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.040843 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jun 15 05:53:39.040867 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.040857 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jun 15 05:53:39.041139 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.041122 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jun 15 05:53:39.041288 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.041271 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jun 15 05:53:39.041678 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.041660 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jun 15 05:53:39.041775 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.041684 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jun 15 05:53:39.041775 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.041720 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jun 15 05:53:39.053238 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.053217 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484"] Jun 15 05:53:39.148259 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.148229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bmv\" (UniqueName: \"kubernetes.io/projected/ede6aba8-4fbf-475b-8822-89fdeef217a4-kube-api-access-59bmv\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.148464 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.148279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.148464 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.148349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.148464 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.148405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6aba8-4fbf-475b-8822-89fdeef217a4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.148614 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.148471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-ca\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.148614 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.148510 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-hub\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.248962 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.248922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-ca\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.249165 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.248970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-hub\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.249165 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.249018 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bmv\" (UniqueName: \"kubernetes.io/projected/ede6aba8-4fbf-475b-8822-89fdeef217a4-kube-api-access-59bmv\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.249165 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.249061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.249165 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.249090 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.249165 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.249133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6aba8-4fbf-475b-8822-89fdeef217a4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.249968 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.249938 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede6aba8-4fbf-475b-8822-89fdeef217a4-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.251638 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.251611 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-ca\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.251744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.251635 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-hub\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.251744 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.251729 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.251853 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.251753 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ede6aba8-4fbf-475b-8822-89fdeef217a4-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.260507 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.260482 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bmv\" (UniqueName: \"kubernetes.io/projected/ede6aba8-4fbf-475b-8822-89fdeef217a4-kube-api-access-59bmv\") pod \"cluster-proxy-proxy-agent-58bc49799-dp484\" (UID: \"ede6aba8-4fbf-475b-8822-89fdeef217a4\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.363236 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.363154 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" Jun 15 05:53:39.494429 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.494403 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484"] Jun 15 05:53:39.496848 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:53:39.496823 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede6aba8_4fbf_475b_8822_89fdeef217a4.slice/crio-f809b06dcf19f48248e6bacaec0a9df3c3b8f6b0bfec34fae9f1c906df7aa235 WatchSource:0}: Error finding container f809b06dcf19f48248e6bacaec0a9df3c3b8f6b0bfec34fae9f1c906df7aa235: Status 404 returned error can't find the container with id f809b06dcf19f48248e6bacaec0a9df3c3b8f6b0bfec34fae9f1c906df7aa235 Jun 15 05:53:39.842792 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:39.842749 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:53:40.361685 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:40.361620 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" event={"ID":"ede6aba8-4fbf-475b-8822-89fdeef217a4","Type":"ContainerStarted","Data":"f809b06dcf19f48248e6bacaec0a9df3c3b8f6b0bfec34fae9f1c906df7aa235"} Jun 15 05:53:42.002479 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:42.002437 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:53:42.002970 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:42.002718 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:53:42.014053 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:42.014028 2569 kubelet.go:1648] "Image garbage collection succeeded" Jun 15 05:53:43.377949 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:43.377920 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" event={"ID":"ede6aba8-4fbf-475b-8822-89fdeef217a4","Type":"ContainerStarted","Data":"44db9f57cb9a02dff8447a67a6e5a6a2819548f16ab99ad3ef322e9450c789c4"} Jun 15 05:53:43.378340 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:43.377969 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" event={"ID":"ede6aba8-4fbf-475b-8822-89fdeef217a4","Type":"ContainerStarted","Data":"88fdcdcd44ae8ddf55e92431eb01fbd4fb53adb5b6bb3b4b9d0fbd51c7e09f8f"} Jun 15 05:53:44.384321 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:44.384285 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" event={"ID":"ede6aba8-4fbf-475b-8822-89fdeef217a4","Type":"ContainerStarted","Data":"b06e94b44d94f735d4a6cae3a772681fce5747679685c688aaeb15cb378a865b"} Jun 15 05:53:44.409282 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:53:44.409227 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58bc49799-dp484" podStartSLOduration=1.7095338949999999 podStartE2EDuration="5.409211321s" totalImagesPullingTime="3.699677426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:53:44.407852208 +0000 UTC m=+302.753055350" watchObservedRunningTime="2026-06-15 05:53:44.409211321 +0000 UTC m=+302.754414472" Jun 15 05:54:00.320517 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:00.320481 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:00.420551 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:00.420505 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:06.699432 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:06.699393 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:06.810437 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:06.810404 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:07.047427 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:07.047396 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:07.225840 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:07.225806 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:07.326628 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:07.326527 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:07.407400 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:07.407360 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:07.478946 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:07.478913 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:07.583913 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:07.583809 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:07.941498 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:07.941406 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:08.352650 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:08.352613 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:08.638671 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:08.638589 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:08.837160 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:08.837130 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:32.776112 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:32.776031 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:32.876244 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:32.876213 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:54:42.272753 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:54:42.272716 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:16.924340 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:16.924266 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:17.036862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:17.036830 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:17.114692 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:17.114657 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:17.186137 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:17.186049 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:17.259926 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:17.259887 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:17.835440 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:17.835410 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:17.934091 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:17.934057 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:17.991391 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:17.991346 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:52.884751 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:52.884717 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:55:52.981403 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:55:52.981368 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:24.366903 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:24.366867 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:25.424818 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:25.424779 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:28.174992 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:28.174953 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:28.269697 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:28.269654 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:29.441343 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:29.441310 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:29.552783 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:29.552749 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:29.745982 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:29.745896 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:32.808850 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:32.808817 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:33.147838 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:33.147751 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:33.245376 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:33.245343 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:33.392369 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:33.392335 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:34.890830 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:34.890798 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:34.979854 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:34.979818 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:35.044442 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:35.044412 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:41.152626 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:41.152587 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:41.288155 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:41.288119 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:41.389363 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:41.389329 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:42.433915 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.433882 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-59c77684cc-hjc4n"] Jun 15 05:56:42.437107 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.437088 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.440575 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.440525 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8vhh4\"" Jun 15 05:56:42.440757 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.440740 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jun 15 05:56:42.441121 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.441106 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jun 15 05:56:42.449882 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.449855 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-59c77684cc-hjc4n"] Jun 15 05:56:42.528422 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.528383 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49k9b\" (UniqueName: \"kubernetes.io/projected/13086940-aea4-465f-b225-c10ac8d2b024-kube-api-access-49k9b\") pod \"cert-manager-59c77684cc-hjc4n\" (UID: \"13086940-aea4-465f-b225-c10ac8d2b024\") " pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.528634 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.528452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13086940-aea4-465f-b225-c10ac8d2b024-bound-sa-token\") pod \"cert-manager-59c77684cc-hjc4n\" (UID: \"13086940-aea4-465f-b225-c10ac8d2b024\") " pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.629883 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.629850 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49k9b\" (UniqueName: \"kubernetes.io/projected/13086940-aea4-465f-b225-c10ac8d2b024-kube-api-access-49k9b\") pod \"cert-manager-59c77684cc-hjc4n\" (UID: \"13086940-aea4-465f-b225-c10ac8d2b024\") " pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.630062 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.629917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13086940-aea4-465f-b225-c10ac8d2b024-bound-sa-token\") pod \"cert-manager-59c77684cc-hjc4n\" (UID: \"13086940-aea4-465f-b225-c10ac8d2b024\") " pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.638790 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.638763 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13086940-aea4-465f-b225-c10ac8d2b024-bound-sa-token\") pod \"cert-manager-59c77684cc-hjc4n\" (UID: \"13086940-aea4-465f-b225-c10ac8d2b024\") " pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.639218 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.639196 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49k9b\" (UniqueName: \"kubernetes.io/projected/13086940-aea4-465f-b225-c10ac8d2b024-kube-api-access-49k9b\") pod \"cert-manager-59c77684cc-hjc4n\" (UID: \"13086940-aea4-465f-b225-c10ac8d2b024\") " pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.710389 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.710300 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:42.746398 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.746358 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-59c77684cc-hjc4n" Jun 15 05:56:42.765426 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.765393 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:42.879766 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.879741 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-59c77684cc-hjc4n"] Jun 15 05:56:42.882047 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:56:42.882020 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13086940_aea4_465f_b225_c10ac8d2b024.slice/crio-92b3f2eb1844564f012e22cc7648540322582089a2b58f0f27fe8e9191243ff9 WatchSource:0}: Error finding container 92b3f2eb1844564f012e22cc7648540322582089a2b58f0f27fe8e9191243ff9: Status 404 returned error can't find the container with id 92b3f2eb1844564f012e22cc7648540322582089a2b58f0f27fe8e9191243ff9 Jun 15 05:56:42.883832 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:42.883813 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 15 05:56:43.066982 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:43.066932 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-59c77684cc-hjc4n" event={"ID":"13086940-aea4-465f-b225-c10ac8d2b024","Type":"ContainerStarted","Data":"92b3f2eb1844564f012e22cc7648540322582089a2b58f0f27fe8e9191243ff9"} Jun 15 05:56:46.080568 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:46.080517 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-59c77684cc-hjc4n" event={"ID":"13086940-aea4-465f-b225-c10ac8d2b024","Type":"ContainerStarted","Data":"559fd0e062862120c5a8f6a90d282ad7d4cb501d3eb39e5220af5a8023f162ef"} Jun 15 05:56:46.098947 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:46.098894 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-59c77684cc-hjc4n" podStartSLOduration=1.5179224150000001 podStartE2EDuration="4.09887536s" totalImagesPullingTime="2.580952945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:46.097861605 +0000 UTC m=+484.443064770" watchObservedRunningTime="2026-06-15 05:56:46.09887536 +0000 UTC m=+484.444078501" Jun 15 05:56:46.524683 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:46.524596 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:46.624813 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:46.624782 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:53.391994 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:53.391954 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:53.492954 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:53.492919 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:53.551701 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:53.551672 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:56:58.438045 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:56:58.438013 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:01.977685 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:01.977646 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:02.032107 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:02.032078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:13.414209 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:13.414179 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:13.515818 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:13.515782 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:14.688311 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:14.688274 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:14.785462 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:14.785423 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:15.145869 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.145833 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:15.312803 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.312774 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:15.415777 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.415648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:15.447994 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.447963 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2"] Jun 15 05:57:15.451116 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.451094 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.464969 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.464948 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jun 15 05:57:15.465905 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.465883 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jun 15 05:57:15.466005 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.465883 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4hzvt\"" Jun 15 05:57:15.466005 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.465884 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jun 15 05:57:15.466005 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.465883 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jun 15 05:57:15.466005 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.465886 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jun 15 05:57:15.492922 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.492893 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2"] Jun 15 05:57:15.516285 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.516246 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9pm\" (UniqueName: \"kubernetes.io/projected/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-kube-api-access-tx9pm\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.516285 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.516291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-cert\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.516485 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.516309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-manager-config\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.516485 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.516337 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-metrics-cert\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.617480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.617443 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9pm\" (UniqueName: \"kubernetes.io/projected/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-kube-api-access-tx9pm\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.617480 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.617481 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-cert\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.617732 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.617500 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-manager-config\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.617732 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.617520 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-metrics-cert\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.618180 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.618148 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-manager-config\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.620078 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.620052 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-cert\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.620173 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.620113 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-metrics-cert\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.629882 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.629852 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9pm\" (UniqueName: \"kubernetes.io/projected/bb11ea56-a48d-4258-ab94-d664c9ceb3f8-kube-api-access-tx9pm\") pod \"lws-controller-manager-74f47976d9-v6qw2\" (UID: \"bb11ea56-a48d-4258-ab94-d664c9ceb3f8\") " pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.759464 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.759380 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:15.821411 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.821359 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:15.909789 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:15.909754 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2"] Jun 15 05:57:15.913956 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:57:15.913934 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb11ea56_a48d_4258_ab94_d664c9ceb3f8.slice/crio-be05db4eddc9da193a240dfeb9cf905c2849118ed786fb5c534e63526f740a2f WatchSource:0}: Error finding container be05db4eddc9da193a240dfeb9cf905c2849118ed786fb5c534e63526f740a2f: Status 404 returned error can't find the container with id be05db4eddc9da193a240dfeb9cf905c2849118ed786fb5c534e63526f740a2f Jun 15 05:57:16.197763 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:16.197728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" event={"ID":"bb11ea56-a48d-4258-ab94-d664c9ceb3f8","Type":"ContainerStarted","Data":"be05db4eddc9da193a240dfeb9cf905c2849118ed786fb5c534e63526f740a2f"} Jun 15 05:57:19.210615 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:19.210581 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" event={"ID":"bb11ea56-a48d-4258-ab94-d664c9ceb3f8","Type":"ContainerStarted","Data":"3e710f29afdaed7835243c0aefe6b99477278332a7d881b54131b0f378c69de5"} Jun 15 05:57:19.211117 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:19.210691 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:19.230872 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:19.230824 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" podStartSLOduration=1.893343792 podStartE2EDuration="4.230809214s" totalImagesPullingTime="2.337465422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:19.229130246 +0000 UTC m=+517.574333387" watchObservedRunningTime="2026-06-15 05:57:19.230809214 +0000 UTC m=+517.576012355" Jun 15 05:57:21.819617 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:21.819568 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:25.716681 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:25.716643 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:25.789639 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:25.789601 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:26.125824 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:26.125775 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:26.228962 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:26.228929 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:26.333637 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:26.333604 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:29.234676 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:29.234640 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:29.335638 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:29.335598 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:30.217242 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:30.217207 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74f47976d9-v6qw2" Jun 15 05:57:33.465992 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:33.465956 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:33.569173 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:33.569142 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:36.577387 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:36.577346 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:40.931099 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:40.931064 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:41.031322 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:41.031287 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:56.218644 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:56.218600 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:56.661170 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:56.661134 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:56.759080 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:56.759043 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:57:57.023728 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:57:57.023647 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:00.068024 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:00.067992 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:00.240998 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:00.240963 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:01.050159 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.050125 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:01.492061 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.491959 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:01.947008 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.946970 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx"] Jun 15 05:58:01.953712 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.953680 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" Jun 15 05:58:01.961829 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.961805 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jun 15 05:58:01.962612 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.962588 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jun 15 05:58:01.962743 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.962643 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bbb69\"" Jun 15 05:58:01.968895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.968877 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jun 15 05:58:01.976352 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:01.976328 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx"] Jun 15 05:58:02.025475 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:02.025442 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/6e5a3783-7faf-453d-92d5-cf282e5dc44e-kube-api-access-p687f\") pod \"dns-operator-controller-manager-65b49595d7-knbkx\" (UID: \"6e5a3783-7faf-453d-92d5-cf282e5dc44e\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" Jun 15 05:58:02.126345 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:02.126314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/6e5a3783-7faf-453d-92d5-cf282e5dc44e-kube-api-access-p687f\") pod \"dns-operator-controller-manager-65b49595d7-knbkx\" (UID: \"6e5a3783-7faf-453d-92d5-cf282e5dc44e\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" Jun 15 05:58:02.139779 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:02.139752 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p687f\" (UniqueName: \"kubernetes.io/projected/6e5a3783-7faf-453d-92d5-cf282e5dc44e-kube-api-access-p687f\") pod \"dns-operator-controller-manager-65b49595d7-knbkx\" (UID: \"6e5a3783-7faf-453d-92d5-cf282e5dc44e\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" Jun 15 05:58:02.246060 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:02.245985 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:02.264340 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:02.264317 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" Jun 15 05:58:02.420275 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:02.420249 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx"] Jun 15 05:58:02.421102 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:58:02.421066 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5a3783_7faf_453d_92d5_cf282e5dc44e.slice/crio-08ec713fc99e6eb41fbdfef1836533a44c3259729baf650fa605e5a00255071c WatchSource:0}: Error finding container 08ec713fc99e6eb41fbdfef1836533a44c3259729baf650fa605e5a00255071c: Status 404 returned error can't find the container with id 08ec713fc99e6eb41fbdfef1836533a44c3259729baf650fa605e5a00255071c Jun 15 05:58:02.683169 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:02.683137 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:03.039691 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:03.039664 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:03.286243 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:03.286214 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:03.390658 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:03.390582 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:03.392848 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:03.392823 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" event={"ID":"6e5a3783-7faf-453d-92d5-cf282e5dc44e","Type":"ContainerStarted","Data":"08ec713fc99e6eb41fbdfef1836533a44c3259729baf650fa605e5a00255071c"} Jun 15 05:58:04.398040 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:04.398004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" event={"ID":"6e5a3783-7faf-453d-92d5-cf282e5dc44e","Type":"ContainerStarted","Data":"bf84782c4d5802dc9f989b428fad86194e95e9bd098e25fcff6c4527b628233f"} Jun 15 05:58:04.398390 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:04.398130 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" Jun 15 05:58:04.419397 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:04.419345 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" podStartSLOduration=1.538118264 podStartE2EDuration="3.419330769s" totalImagesPullingTime="1.881212505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:04.418319269 +0000 UTC m=+562.763522423" watchObservedRunningTime="2026-06-15 05:58:04.419330769 +0000 UTC m=+562.764533909" Jun 15 05:58:05.902004 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:05.901966 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4"] Jun 15 05:58:05.905229 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:05.905211 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:05.907700 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:05.907678 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qd98c\"" Jun 15 05:58:05.918402 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:05.918380 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4"] Jun 15 05:58:05.960210 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:05.960171 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5c9bd5678d-xwtf4\" (UID: \"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:05.960210 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:05.960215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjr6n\" (UniqueName: \"kubernetes.io/projected/75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b-kube-api-access-pjr6n\") pod \"kuadrant-operator-controller-manager-5c9bd5678d-xwtf4\" (UID: \"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:06.060641 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.060609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5c9bd5678d-xwtf4\" (UID: \"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:06.060814 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.060652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjr6n\" (UniqueName: \"kubernetes.io/projected/75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b-kube-api-access-pjr6n\") pod \"kuadrant-operator-controller-manager-5c9bd5678d-xwtf4\" (UID: \"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:06.060972 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.060953 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5c9bd5678d-xwtf4\" (UID: \"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:06.069552 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.069519 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjr6n\" (UniqueName: \"kubernetes.io/projected/75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b-kube-api-access-pjr6n\") pod \"kuadrant-operator-controller-manager-5c9bd5678d-xwtf4\" (UID: \"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:06.215216 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.215131 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:06.352328 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.352193 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4"] Jun 15 05:58:06.354755 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:58:06.354727 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fc04a5_2d46_4445_8d6a_6fbb4e3bc41b.slice/crio-1271bf7c05285c8b5b099255446c6a0a9cbe91aca2b80fac8c8f234afada6889 WatchSource:0}: Error finding container 1271bf7c05285c8b5b099255446c6a0a9cbe91aca2b80fac8c8f234afada6889: Status 404 returned error can't find the container with id 1271bf7c05285c8b5b099255446c6a0a9cbe91aca2b80fac8c8f234afada6889 Jun 15 05:58:06.410695 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.410587 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" event={"ID":"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b","Type":"ContainerStarted","Data":"1271bf7c05285c8b5b099255446c6a0a9cbe91aca2b80fac8c8f234afada6889"} Jun 15 05:58:06.428005 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:06.427956 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:07.896988 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:07.896949 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:09.459895 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:09.459866 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:09.569855 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:09.569823 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:09.845973 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:09.845943 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:09.945303 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:09.945252 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:10.431932 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:10.431837 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" event={"ID":"75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b","Type":"ContainerStarted","Data":"4491c17ad54e8cb426715604e3d13811ceed5164ea2873fed8fe86501714d35f"} Jun 15 05:58:10.431932 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:10.431905 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:10.459211 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:10.459147 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" podStartSLOduration=1.691381012 podStartE2EDuration="5.45913251s" totalImagesPullingTime="3.767751498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:10.458949942 +0000 UTC m=+568.804153079" watchObservedRunningTime="2026-06-15 05:58:10.45913251 +0000 UTC m=+568.804335651" Jun 15 05:58:12.529120 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:12.529089 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:12.711448 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:12.711406 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:13.454682 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:13.454648 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:13.517278 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:13.517245 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:15.405322 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:15.405291 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-knbkx" Jun 15 05:58:19.027591 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:19.027531 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:19.127721 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:19.127677 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:19.232181 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:19.232148 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:21.437933 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:21.437904 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5c9bd5678d-xwtf4" Jun 15 05:58:26.035832 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:26.035791 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:26.148020 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:26.147981 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:29.852993 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:29.852962 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:29.918484 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:29.918428 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:30.453613 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:30.453580 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:30.550381 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:30.550345 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:33.638629 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:33.638585 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:33.749784 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:33.749746 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:38.812670 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:38.812636 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:38.914823 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:38.914788 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:38.991584 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:38.991526 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:42.034649 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:42.034617 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:58:42.035069 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:42.034949 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 05:58:44.973728 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:44.973689 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:45.990828 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:45.990762 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:53.477984 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.477942 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:53.491343 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.491315 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-frnlb"] Jun 15 05:58:53.493621 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.493601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.495963 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.495928 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jun 15 05:58:53.496097 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.496008 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-dvq6x\"" Jun 15 05:58:53.503624 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.503603 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-frnlb"] Jun 15 05:58:53.586316 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.586280 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:53.586316 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.586303 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-frnlb"] Jun 15 05:58:53.595435 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.595410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htcwk\" (UniqueName: \"kubernetes.io/projected/4b419f9a-7d93-4374-b35f-932d45dd1020-kube-api-access-htcwk\") pod \"limitador-limitador-ff68ff5cb-frnlb\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.595580 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.595450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b419f9a-7d93-4374-b35f-932d45dd1020-config-file\") pod \"limitador-limitador-ff68ff5cb-frnlb\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.687758 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.687721 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:53.696857 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.696829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b419f9a-7d93-4374-b35f-932d45dd1020-config-file\") pod \"limitador-limitador-ff68ff5cb-frnlb\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.696994 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.696911 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htcwk\" (UniqueName: \"kubernetes.io/projected/4b419f9a-7d93-4374-b35f-932d45dd1020-kube-api-access-htcwk\") pod \"limitador-limitador-ff68ff5cb-frnlb\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.697487 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.697468 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b419f9a-7d93-4374-b35f-932d45dd1020-config-file\") pod \"limitador-limitador-ff68ff5cb-frnlb\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.706596 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.706572 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htcwk\" (UniqueName: \"kubernetes.io/projected/4b419f9a-7d93-4374-b35f-932d45dd1020-kube-api-access-htcwk\") pod \"limitador-limitador-ff68ff5cb-frnlb\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.805004 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.804966 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:53.950450 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:53.950424 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-frnlb"] Jun 15 05:58:53.952708 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:58:53.952668 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b419f9a_7d93_4374_b35f_932d45dd1020.slice/crio-ad29e148ab8324ae31a8588e4a5aa0582d36cd292dd09980d6685f310eb98403 WatchSource:0}: Error finding container ad29e148ab8324ae31a8588e4a5aa0582d36cd292dd09980d6685f310eb98403: Status 404 returned error can't find the container with id ad29e148ab8324ae31a8588e4a5aa0582d36cd292dd09980d6685f310eb98403 Jun 15 05:58:54.354422 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:54.354376 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:54.437902 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:54.437871 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:58:54.629862 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:54.629783 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" event={"ID":"4b419f9a-7d93-4374-b35f-932d45dd1020","Type":"ContainerStarted","Data":"ad29e148ab8324ae31a8588e4a5aa0582d36cd292dd09980d6685f310eb98403"} Jun 15 05:58:58.653332 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:58.653291 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" event={"ID":"4b419f9a-7d93-4374-b35f-932d45dd1020","Type":"ContainerStarted","Data":"641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11"} Jun 15 05:58:58.653802 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:58.653413 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:58:58.671730 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:58:58.671679 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" podStartSLOduration=1.897461489 podStartE2EDuration="5.671662487s" totalImagesPullingTime="3.774200998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:58.671142996 +0000 UTC m=+617.016346139" watchObservedRunningTime="2026-06-15 05:58:58.671662487 +0000 UTC m=+617.016865664" Jun 15 05:59:09.659336 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:09.659262 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:59:09.845678 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:09.845645 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-frnlb"] Jun 15 05:59:09.845868 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:09.845847 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" podUID="4b419f9a-7d93-4374-b35f-932d45dd1020" containerName="limitador" containerID="cri-o://641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11" gracePeriod=30 Jun 15 05:59:09.967996 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:09.967910 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:10.033418 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.033386 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:10.393745 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.393721 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:59:10.545936 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.545905 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b419f9a-7d93-4374-b35f-932d45dd1020-config-file\") pod \"4b419f9a-7d93-4374-b35f-932d45dd1020\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " Jun 15 05:59:10.546087 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.546038 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htcwk\" (UniqueName: \"kubernetes.io/projected/4b419f9a-7d93-4374-b35f-932d45dd1020-kube-api-access-htcwk\") pod \"4b419f9a-7d93-4374-b35f-932d45dd1020\" (UID: \"4b419f9a-7d93-4374-b35f-932d45dd1020\") " Jun 15 05:59:10.546293 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.546266 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b419f9a-7d93-4374-b35f-932d45dd1020-config-file" (OuterVolumeSpecName: "config-file") pod "4b419f9a-7d93-4374-b35f-932d45dd1020" (UID: "4b419f9a-7d93-4374-b35f-932d45dd1020"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jun 15 05:59:10.548295 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.548261 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b419f9a-7d93-4374-b35f-932d45dd1020-kube-api-access-htcwk" (OuterVolumeSpecName: "kube-api-access-htcwk") pod "4b419f9a-7d93-4374-b35f-932d45dd1020" (UID: "4b419f9a-7d93-4374-b35f-932d45dd1020"). InnerVolumeSpecName "kube-api-access-htcwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 05:59:10.646848 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.646810 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htcwk\" (UniqueName: \"kubernetes.io/projected/4b419f9a-7d93-4374-b35f-932d45dd1020-kube-api-access-htcwk\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:59:10.646848 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.646842 2569 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4b419f9a-7d93-4374-b35f-932d45dd1020-config-file\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 05:59:10.718576 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.715633 2569 generic.go:356] "Generic (PLEG): container finished" podID="4b419f9a-7d93-4374-b35f-932d45dd1020" containerID="641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11" exitCode=0 Jun 15 05:59:10.718576 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.715733 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" event={"ID":"4b419f9a-7d93-4374-b35f-932d45dd1020","Type":"ContainerDied","Data":"641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11"} Jun 15 05:59:10.718576 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.715780 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" event={"ID":"4b419f9a-7d93-4374-b35f-932d45dd1020","Type":"ContainerDied","Data":"ad29e148ab8324ae31a8588e4a5aa0582d36cd292dd09980d6685f310eb98403"} Jun 15 05:59:10.718576 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.715806 2569 scope.go:117] "RemoveContainer" containerID="641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11" Jun 15 05:59:10.718576 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.715958 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-frnlb" Jun 15 05:59:10.730842 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.730758 2569 scope.go:117] "RemoveContainer" containerID="641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11" Jun 15 05:59:10.731269 ip-10-0-128-226 kubenswrapper[2569]: E0615 05:59:10.731203 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11\": container with ID starting with 641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11 not found: ID does not exist" containerID="641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11" Jun 15 05:59:10.731269 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.731234 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11"} err="failed to get container status \"641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11\": rpc error: code = NotFound desc = could not find container \"641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11\": container with ID starting with 641a7f27b21e14225e319908cb3404a37caa504618dd601223ff7868c70b3a11 not found: ID does not exist" Jun 15 05:59:10.748265 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.748233 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-frnlb"] Jun 15 05:59:10.752399 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:10.752376 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-frnlb"] Jun 15 05:59:12.056568 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:12.056523 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b419f9a-7d93-4374-b35f-932d45dd1020" path="/var/lib/kubelet/pods/4b419f9a-7d93-4374-b35f-932d45dd1020/volumes" Jun 15 05:59:12.915691 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:12.915642 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:13.776678 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:13.776644 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:13.836207 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:13.836179 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:15.943045 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:15.942994 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:16.211910 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:16.211814 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:58.790267 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.790221 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:58.831168 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.831138 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-x2wgq"] Jun 15 05:59:58.831457 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.831445 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b419f9a-7d93-4374-b35f-932d45dd1020" containerName="limitador" Jun 15 05:59:58.831510 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.831458 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b419f9a-7d93-4374-b35f-932d45dd1020" containerName="limitador" Jun 15 05:59:58.831564 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.831530 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b419f9a-7d93-4374-b35f-932d45dd1020" containerName="limitador" Jun 15 05:59:58.835720 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.835699 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:58.837948 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.837918 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Jun 15 05:59:58.838077 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.837993 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-8clcc\"" Jun 15 05:59:58.838508 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.838479 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jun 15 05:59:58.838779 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.838763 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jun 15 05:59:58.845780 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.845757 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-x2wgq"] Jun 15 05:59:58.847326 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.847306 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-z4qqr"] Jun 15 05:59:58.850615 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.850595 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:58.852725 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.852708 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\"" Jun 15 05:59:58.852892 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.852876 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-z99z8\"" Jun 15 05:59:58.859259 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.859238 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-z4qqr"] Jun 15 05:59:58.862219 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.862195 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:58.933157 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.933122 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:58.981396 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.981356 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba617900-b001-4d38-8840-74301c5f4a0d-tls-certs\") pod \"model-serving-api-fd65d7d6-x2wgq\" (UID: \"ba617900-b001-4d38-8840-74301c5f4a0d\") " pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:58.981396 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.981403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d4ba447-7c6d-443f-9ac1-e057420d6bfa-cert\") pod \"odh-model-controller-84b54b7f97-z4qqr\" (UID: \"9d4ba447-7c6d-443f-9ac1-e057420d6bfa\") " pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:58.981652 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.981456 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh68z\" (UniqueName: \"kubernetes.io/projected/9d4ba447-7c6d-443f-9ac1-e057420d6bfa-kube-api-access-sh68z\") pod \"odh-model-controller-84b54b7f97-z4qqr\" (UID: \"9d4ba447-7c6d-443f-9ac1-e057420d6bfa\") " pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:58.981652 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:58.981498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdfr5\" (UniqueName: \"kubernetes.io/projected/ba617900-b001-4d38-8840-74301c5f4a0d-kube-api-access-bdfr5\") pod \"model-serving-api-fd65d7d6-x2wgq\" (UID: \"ba617900-b001-4d38-8840-74301c5f4a0d\") " pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:59.082891 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.082807 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d4ba447-7c6d-443f-9ac1-e057420d6bfa-cert\") pod \"odh-model-controller-84b54b7f97-z4qqr\" (UID: \"9d4ba447-7c6d-443f-9ac1-e057420d6bfa\") " pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:59.082891 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.082865 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh68z\" (UniqueName: \"kubernetes.io/projected/9d4ba447-7c6d-443f-9ac1-e057420d6bfa-kube-api-access-sh68z\") pod \"odh-model-controller-84b54b7f97-z4qqr\" (UID: \"9d4ba447-7c6d-443f-9ac1-e057420d6bfa\") " pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:59.083091 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.082904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdfr5\" (UniqueName: \"kubernetes.io/projected/ba617900-b001-4d38-8840-74301c5f4a0d-kube-api-access-bdfr5\") pod \"model-serving-api-fd65d7d6-x2wgq\" (UID: \"ba617900-b001-4d38-8840-74301c5f4a0d\") " pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:59.083091 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.083012 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba617900-b001-4d38-8840-74301c5f4a0d-tls-certs\") pod \"model-serving-api-fd65d7d6-x2wgq\" (UID: \"ba617900-b001-4d38-8840-74301c5f4a0d\") " pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:59.085649 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.085619 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba617900-b001-4d38-8840-74301c5f4a0d-tls-certs\") pod \"model-serving-api-fd65d7d6-x2wgq\" (UID: \"ba617900-b001-4d38-8840-74301c5f4a0d\") " pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:59.085773 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.085620 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d4ba447-7c6d-443f-9ac1-e057420d6bfa-cert\") pod \"odh-model-controller-84b54b7f97-z4qqr\" (UID: \"9d4ba447-7c6d-443f-9ac1-e057420d6bfa\") " pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:59.094184 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.094156 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdfr5\" (UniqueName: \"kubernetes.io/projected/ba617900-b001-4d38-8840-74301c5f4a0d-kube-api-access-bdfr5\") pod \"model-serving-api-fd65d7d6-x2wgq\" (UID: \"ba617900-b001-4d38-8840-74301c5f4a0d\") " pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:59.094320 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.094169 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh68z\" (UniqueName: \"kubernetes.io/projected/9d4ba447-7c6d-443f-9ac1-e057420d6bfa-kube-api-access-sh68z\") pod \"odh-model-controller-84b54b7f97-z4qqr\" (UID: \"9d4ba447-7c6d-443f-9ac1-e057420d6bfa\") " pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:59.121932 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.121899 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:59.148721 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.148695 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 05:59:59.164845 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.164819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 05:59:59.222942 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.222891 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 05:59:59.289964 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.289933 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-x2wgq"] Jun 15 05:59:59.292917 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:59:59.292874 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba617900_b001_4d38_8840_74301c5f4a0d.slice/crio-d2b68ca529e91f53e87e503b68ff7614d3271afa1caa066f24d9d981641bb63b WatchSource:0}: Error finding container d2b68ca529e91f53e87e503b68ff7614d3271afa1caa066f24d9d981641bb63b: Status 404 returned error can't find the container with id d2b68ca529e91f53e87e503b68ff7614d3271afa1caa066f24d9d981641bb63b Jun 15 05:59:59.311416 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.311394 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-z4qqr"] Jun 15 05:59:59.313700 ip-10-0-128-226 kubenswrapper[2569]: W0615 05:59:59.313675 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d4ba447_7c6d_443f_9ac1_e057420d6bfa.slice/crio-b7144d54c591fa7da36a531001cef2b4423bae2a1ea3a9072a45c6960f683980 WatchSource:0}: Error finding container b7144d54c591fa7da36a531001cef2b4423bae2a1ea3a9072a45c6960f683980: Status 404 returned error can't find the container with id b7144d54c591fa7da36a531001cef2b4423bae2a1ea3a9072a45c6960f683980 Jun 15 05:59:59.927154 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.927115 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-z4qqr" event={"ID":"9d4ba447-7c6d-443f-9ac1-e057420d6bfa","Type":"ContainerStarted","Data":"b7144d54c591fa7da36a531001cef2b4423bae2a1ea3a9072a45c6960f683980"} Jun 15 05:59:59.930085 ip-10-0-128-226 kubenswrapper[2569]: I0615 05:59:59.929978 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-x2wgq" event={"ID":"ba617900-b001-4d38-8840-74301c5f4a0d","Type":"ContainerStarted","Data":"d2b68ca529e91f53e87e503b68ff7614d3271afa1caa066f24d9d981641bb63b"} Jun 15 06:00:00.937819 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:00.937719 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-x2wgq" event={"ID":"ba617900-b001-4d38-8840-74301c5f4a0d","Type":"ContainerStarted","Data":"18f479a90fdbf2f561f7d1bc7f1b6a708d1fd2d01ded718fece52fb9d86aabcc"} Jun 15 06:00:00.937819 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:00.937807 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 06:00:00.958494 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:00.958432 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-x2wgq" podStartSLOduration=1.600739377 podStartE2EDuration="2.958416965s" totalImagesPullingTime="1.357677588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:00:00.956246603 +0000 UTC m=+679.301449745" watchObservedRunningTime="2026-06-15 06:00:00.958416965 +0000 UTC m=+679.303620106" Jun 15 06:00:02.955604 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:02.955562 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-z4qqr" event={"ID":"9d4ba447-7c6d-443f-9ac1-e057420d6bfa","Type":"ContainerStarted","Data":"60ea951646418994f67850684b39e55e83054addafb9a83481f96cba70a10ed7"} Jun 15 06:00:02.956079 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:02.955686 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 06:00:02.972931 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:02.972879 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-z4qqr" podStartSLOduration=2.232371856 podStartE2EDuration="4.972863294s" totalImagesPullingTime="2.740491438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:00:02.971905018 +0000 UTC m=+681.317108160" watchObservedRunningTime="2026-06-15 06:00:02.972863294 +0000 UTC m=+681.318066434" Jun 15 06:00:11.947450 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:11.947421 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-x2wgq" Jun 15 06:00:13.962892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:13.962857 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-z4qqr" Jun 15 06:00:14.781249 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.781202 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rpbr4"] Jun 15 06:00:14.784509 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.784493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpbr4" Jun 15 06:00:14.786820 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.786803 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Jun 15 06:00:14.786941 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.786926 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-k7mxw\"" Jun 15 06:00:14.792988 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.792963 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rpbr4"] Jun 15 06:00:14.822714 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.822682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5stxr\" (UniqueName: \"kubernetes.io/projected/58933d47-3efe-4d9d-96be-e807e88f213e-kube-api-access-5stxr\") pod \"s3-init-rpbr4\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " pod="kserve/s3-init-rpbr4" Jun 15 06:00:14.822843 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.822719 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58933d47-3efe-4d9d-96be-e807e88f213e-model-cache\") pod \"s3-init-rpbr4\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " pod="kserve/s3-init-rpbr4" Jun 15 06:00:14.924103 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.924062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5stxr\" (UniqueName: \"kubernetes.io/projected/58933d47-3efe-4d9d-96be-e807e88f213e-kube-api-access-5stxr\") pod \"s3-init-rpbr4\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " pod="kserve/s3-init-rpbr4" Jun 15 06:00:14.924276 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.924111 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58933d47-3efe-4d9d-96be-e807e88f213e-model-cache\") pod \"s3-init-rpbr4\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " pod="kserve/s3-init-rpbr4" Jun 15 06:00:14.924507 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.924484 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58933d47-3efe-4d9d-96be-e807e88f213e-model-cache\") pod \"s3-init-rpbr4\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " pod="kserve/s3-init-rpbr4" Jun 15 06:00:14.933698 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:14.933675 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5stxr\" (UniqueName: \"kubernetes.io/projected/58933d47-3efe-4d9d-96be-e807e88f213e-kube-api-access-5stxr\") pod \"s3-init-rpbr4\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " pod="kserve/s3-init-rpbr4" Jun 15 06:00:15.094565 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:15.094514 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpbr4" Jun 15 06:00:15.225489 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:15.225462 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rpbr4"] Jun 15 06:00:15.227378 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:00:15.227342 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58933d47_3efe_4d9d_96be_e807e88f213e.slice/crio-500088227ff6f6dd6e55e9dd9ab3a7dc0d2cb522950e977d64d65960ed4b30f9 WatchSource:0}: Error finding container 500088227ff6f6dd6e55e9dd9ab3a7dc0d2cb522950e977d64d65960ed4b30f9: Status 404 returned error can't find the container with id 500088227ff6f6dd6e55e9dd9ab3a7dc0d2cb522950e977d64d65960ed4b30f9 Jun 15 06:00:16.014347 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:16.014302 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpbr4" event={"ID":"58933d47-3efe-4d9d-96be-e807e88f213e","Type":"ContainerStarted","Data":"500088227ff6f6dd6e55e9dd9ab3a7dc0d2cb522950e977d64d65960ed4b30f9"} Jun 15 06:00:18.025606 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:18.025578 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpbr4" event={"ID":"58933d47-3efe-4d9d-96be-e807e88f213e","Type":"ContainerStarted","Data":"d7245033c8d049e74aeba719febe8b46ffa19ee9538c641f2c47474b8d76cc96"} Jun 15 06:00:20.040469 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:20.040433 2569 generic.go:356] "Generic (PLEG): container finished" podID="58933d47-3efe-4d9d-96be-e807e88f213e" containerID="d7245033c8d049e74aeba719febe8b46ffa19ee9538c641f2c47474b8d76cc96" exitCode=0 Jun 15 06:00:20.040875 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:20.040511 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpbr4" event={"ID":"58933d47-3efe-4d9d-96be-e807e88f213e","Type":"ContainerDied","Data":"d7245033c8d049e74aeba719febe8b46ffa19ee9538c641f2c47474b8d76cc96"} Jun 15 06:00:22.012934 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:22.012899 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:00:22.113137 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:22.113094 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:00:25.070726 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:25.070693 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpbr4" event={"ID":"58933d47-3efe-4d9d-96be-e807e88f213e","Type":"ContainerStarted","Data":"2171e2f79bbeb85498e18c94f42bd219b88dddff8404f5360e00cffa46e7254f"} Jun 15 06:00:25.088893 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:00:25.088813 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-rpbr4" podStartSLOduration=2.598659297 podStartE2EDuration="11.088796945s" totalImagesPullingTime="7.155701099s" totalInitContainerRuntime="1.334436549s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:00:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:00:25.088470209 +0000 UTC m=+703.433673378" watchObservedRunningTime="2026-06-15 06:00:25.088796945 +0000 UTC m=+703.434000085" Jun 15 06:01:30.358126 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:30.358090 2569 generic.go:356] "Generic (PLEG): container finished" podID="58933d47-3efe-4d9d-96be-e807e88f213e" containerID="2171e2f79bbeb85498e18c94f42bd219b88dddff8404f5360e00cffa46e7254f" exitCode=0 Jun 15 06:01:30.358698 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:30.358163 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpbr4" event={"ID":"58933d47-3efe-4d9d-96be-e807e88f213e","Type":"ContainerDied","Data":"2171e2f79bbeb85498e18c94f42bd219b88dddff8404f5360e00cffa46e7254f"} Jun 15 06:01:31.488996 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:31.488965 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpbr4" Jun 15 06:01:31.511127 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:31.511099 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5stxr\" (UniqueName: \"kubernetes.io/projected/58933d47-3efe-4d9d-96be-e807e88f213e-kube-api-access-5stxr\") pod \"58933d47-3efe-4d9d-96be-e807e88f213e\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " Jun 15 06:01:31.511290 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:31.511136 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58933d47-3efe-4d9d-96be-e807e88f213e-model-cache\") pod \"58933d47-3efe-4d9d-96be-e807e88f213e\" (UID: \"58933d47-3efe-4d9d-96be-e807e88f213e\") " Jun 15 06:01:31.513269 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:31.513236 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58933d47-3efe-4d9d-96be-e807e88f213e-kube-api-access-5stxr" (OuterVolumeSpecName: "kube-api-access-5stxr") pod "58933d47-3efe-4d9d-96be-e807e88f213e" (UID: "58933d47-3efe-4d9d-96be-e807e88f213e"). InnerVolumeSpecName "kube-api-access-5stxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:01:31.530479 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:31.530431 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58933d47-3efe-4d9d-96be-e807e88f213e-model-cache" (OuterVolumeSpecName: "model-cache") pod "58933d47-3efe-4d9d-96be-e807e88f213e" (UID: "58933d47-3efe-4d9d-96be-e807e88f213e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:01:31.612084 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:31.612044 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5stxr\" (UniqueName: \"kubernetes.io/projected/58933d47-3efe-4d9d-96be-e807e88f213e-kube-api-access-5stxr\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:01:31.612084 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:31.612078 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58933d47-3efe-4d9d-96be-e807e88f213e-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:01:32.368321 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:32.368242 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rpbr4" Jun 15 06:01:32.368321 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:32.368246 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rpbr4" event={"ID":"58933d47-3efe-4d9d-96be-e807e88f213e","Type":"ContainerDied","Data":"500088227ff6f6dd6e55e9dd9ab3a7dc0d2cb522950e977d64d65960ed4b30f9"} Jun 15 06:01:32.368321 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:32.368286 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="500088227ff6f6dd6e55e9dd9ab3a7dc0d2cb522950e977d64d65960ed4b30f9" Jun 15 06:01:32.753718 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:32.753634 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:01:32.857451 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:32.857419 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:01:44.855450 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.855414 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf"] Jun 15 06:01:44.855813 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.855775 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58933d47-3efe-4d9d-96be-e807e88f213e" containerName="s3-init" Jun 15 06:01:44.855813 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.855789 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58933d47-3efe-4d9d-96be-e807e88f213e" containerName="s3-init" Jun 15 06:01:44.855813 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.855803 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58933d47-3efe-4d9d-96be-e807e88f213e" containerName="download-hf-model" Jun 15 06:01:44.855813 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.855809 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="58933d47-3efe-4d9d-96be-e807e88f213e" containerName="download-hf-model" Jun 15 06:01:44.855960 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.855870 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="58933d47-3efe-4d9d-96be-e807e88f213e" containerName="s3-init" Jun 15 06:01:44.857749 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.857731 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.864303 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.864241 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jun 15 06:01:44.864446 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.864426 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\"" Jun 15 06:01:44.864697 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.864679 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jun 15 06:01:44.865229 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.865209 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-79qdt\"" Jun 15 06:01:44.878936 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.878911 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:01:44.881476 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.881451 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf"] Jun 15 06:01:44.933925 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.933884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-workload-socket\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.933925 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.933927 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-envoy\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.934191 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.933994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-token\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.934191 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.934062 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-podinfo\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.934191 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.934137 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-credential-socket\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.934191 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.934172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-data\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.934358 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.934247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh29q\" (UniqueName: \"kubernetes.io/projected/e8405d0a-7af6-4f75-9d69-2a63b8579195-kube-api-access-gh29q\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.934358 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.934306 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8405d0a-7af6-4f75-9d69-2a63b8579195-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.934358 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.934334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-workload-certs\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:44.977595 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:44.977561 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:01:45.035352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-credential-socket\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035363 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-data\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035615 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035403 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh29q\" (UniqueName: \"kubernetes.io/projected/e8405d0a-7af6-4f75-9d69-2a63b8579195-kube-api-access-gh29q\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035615 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035573 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8405d0a-7af6-4f75-9d69-2a63b8579195-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035726 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035619 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-workload-certs\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035726 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035662 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-workload-socket\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035726 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035692 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-envoy\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035881 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-token\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035881 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035774 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-podinfo\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.035881 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.035799 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-credential-socket\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.036073 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.036044 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-workload-socket\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.036199 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.036154 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-data\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.036386 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.036366 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-workload-certs\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.036626 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.036608 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8405d0a-7af6-4f75-9d69-2a63b8579195-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.038561 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.038522 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-envoy\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.038843 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.038823 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-podinfo\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.044642 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.044622 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8405d0a-7af6-4f75-9d69-2a63b8579195-istio-token\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.044837 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.044813 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh29q\" (UniqueName: \"kubernetes.io/projected/e8405d0a-7af6-4f75-9d69-2a63b8579195-kube-api-access-gh29q\") pod \"router-gateway-1-openshift-default-75dcfd69c9-dh6qf\" (UID: \"e8405d0a-7af6-4f75-9d69-2a63b8579195\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.168147 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.168050 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:45.308894 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.308765 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf"] Jun 15 06:01:45.311865 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:01:45.311838 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8405d0a_7af6_4f75_9d69_2a63b8579195.slice/crio-41336d44d981927006a957ba0e0c19cb8ced66b58511e1e8efbb5752a679a146 WatchSource:0}: Error finding container 41336d44d981927006a957ba0e0c19cb8ced66b58511e1e8efbb5752a679a146: Status 404 returned error can't find the container with id 41336d44d981927006a957ba0e0c19cb8ced66b58511e1e8efbb5752a679a146 Jun 15 06:01:45.314294 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.314278 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 15 06:01:45.429929 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:45.429840 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" event={"ID":"e8405d0a-7af6-4f75-9d69-2a63b8579195","Type":"ContainerStarted","Data":"41336d44d981927006a957ba0e0c19cb8ced66b58511e1e8efbb5752a679a146"} Jun 15 06:01:48.021937 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:48.021880 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jun 15 06:01:48.022234 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:48.021979 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jun 15 06:01:48.022234 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:48.022009 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jun 15 06:01:48.449409 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:48.449371 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" event={"ID":"e8405d0a-7af6-4f75-9d69-2a63b8579195","Type":"ContainerStarted","Data":"faa715e63cf60dc52c97a5dd0eb4edf9de7769d6139b4438d59a54d813cf808e"} Jun 15 06:01:48.472023 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:48.471974 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" podStartSLOduration=1.7647025859999999 podStartE2EDuration="4.471959255s" totalImagesPullingTime="2.707256669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:01:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:01:48.469801823 +0000 UTC m=+786.815004955" watchObservedRunningTime="2026-06-15 06:01:48.471959255 +0000 UTC m=+786.817162396" Jun 15 06:01:48.655781 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:48.655745 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:01:49.168775 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:49.168737 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:49.173697 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:49.173668 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:49.454906 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:49.454820 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:49.455843 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:49.455825 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:01:49.855373 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:49.855335 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:01:50.223533 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.223459 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf"] Jun 15 06:01:50.225985 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.225960 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.229090 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.229064 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\"" Jun 15 06:01:50.229237 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.229088 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fjfwp\"" Jun 15 06:01:50.229237 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.229070 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-ns4gl\"" Jun 15 06:01:50.241915 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.241887 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf"] Jun 15 06:01:50.252138 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.252106 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:01:50.390632 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.390598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.390632 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.390634 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tspf\" (UniqueName: \"kubernetes.io/projected/a3814454-7fca-44a2-a75f-ca9e86bba7e2-kube-api-access-4tspf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.390902 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.390709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.390902 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.390725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.390902 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.390826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491512 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491512 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491458 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491512 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491830 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491529 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491830 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tspf\" (UniqueName: \"kubernetes.io/projected/a3814454-7fca-44a2-a75f-ca9e86bba7e2-kube-api-access-4tspf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491941 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491898 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491999 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491949 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.491999 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.491979 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.494141 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.494110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.501057 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.501035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tspf\" (UniqueName: \"kubernetes.io/projected/a3814454-7fca-44a2-a75f-ca9e86bba7e2-kube-api-access-4tspf\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.536803 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.536769 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:01:50.679103 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:50.679077 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf"] Jun 15 06:01:50.681168 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:01:50.681132 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3814454_7fca_44a2_a75f_ca9e86bba7e2.slice/crio-c4421d51c2d40462749cf03ea5c5f689489bc4f39ad2daf48a52e458cce0064a WatchSource:0}: Error finding container c4421d51c2d40462749cf03ea5c5f689489bc4f39ad2daf48a52e458cce0064a: Status 404 returned error can't find the container with id c4421d51c2d40462749cf03ea5c5f689489bc4f39ad2daf48a52e458cce0064a Jun 15 06:01:51.466007 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:51.465963 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" event={"ID":"a3814454-7fca-44a2-a75f-ca9e86bba7e2","Type":"ContainerStarted","Data":"c4421d51c2d40462749cf03ea5c5f689489bc4f39ad2daf48a52e458cce0064a"} Jun 15 06:01:52.474808 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:01:52.474762 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" event={"ID":"a3814454-7fca-44a2-a75f-ca9e86bba7e2","Type":"ContainerStarted","Data":"141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d"} Jun 15 06:02:14.513446 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:14.513408 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:02:16.885039 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:16.884997 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:02:22.655774 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:22.655685 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" event={"ID":"a3814454-7fca-44a2-a75f-ca9e86bba7e2","Type":"ContainerStarted","Data":"58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa"} Jun 15 06:02:22.656326 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:22.656304 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:02:22.658508 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:22.658485 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:02:22.676913 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:22.676844 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" podStartSLOduration=1.132868523 podStartE2EDuration="32.676819899s" totalImagesPullingTime="31.543951376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:01:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:02:22.675953825 +0000 UTC m=+821.021156965" watchObservedRunningTime="2026-06-15 06:02:22.676819899 +0000 UTC m=+821.022023051" Jun 15 06:02:27.683874 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:27.683783 2569 generic.go:356] "Generic (PLEG): container finished" podID="e8405d0a-7af6-4f75-9d69-2a63b8579195" containerID="faa715e63cf60dc52c97a5dd0eb4edf9de7769d6139b4438d59a54d813cf808e" exitCode=0 Jun 15 06:02:27.683874 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:27.683855 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" event={"ID":"e8405d0a-7af6-4f75-9d69-2a63b8579195","Type":"ContainerDied","Data":"faa715e63cf60dc52c97a5dd0eb4edf9de7769d6139b4438d59a54d813cf808e"} Jun 15 06:02:27.684313 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:27.684290 2569 scope.go:117] "RemoveContainer" containerID="faa715e63cf60dc52c97a5dd0eb4edf9de7769d6139b4438d59a54d813cf808e" Jun 15 06:02:27.685285 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:27.685250 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jun 15 06:02:27.685366 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:27.685340 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jun 15 06:02:27.685403 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:27.685376 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jun 15 06:02:28.690685 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:28.690648 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" event={"ID":"e8405d0a-7af6-4f75-9d69-2a63b8579195","Type":"ContainerStarted","Data":"cb7cfe81481c68cb690261e8074283fdabc51ada243b8bad23f546c4dcf208bf"} Jun 15 06:02:29.169107 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:29.169068 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:02:29.169296 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:29.169261 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" podUID="e8405d0a-7af6-4f75-9d69-2a63b8579195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jun 15 06:02:30.168705 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:30.168655 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" podUID="e8405d0a-7af6-4f75-9d69-2a63b8579195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jun 15 06:02:30.537352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:30.537316 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:02:30.537577 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:30.537399 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:02:30.539301 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:30.539269 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:02:30.701054 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:30.701009 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:02:31.169239 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:31.169195 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" podUID="e8405d0a-7af6-4f75-9d69-2a63b8579195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jun 15 06:02:32.169418 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:32.169382 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" podUID="e8405d0a-7af6-4f75-9d69-2a63b8579195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jun 15 06:02:33.169471 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:33.169429 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" podUID="e8405d0a-7af6-4f75-9d69-2a63b8579195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.28:15021/healthz/ready\": dial tcp 10.134.0.28:15021: connect: connection refused" Jun 15 06:02:34.172812 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:34.172776 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:02:34.173224 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:34.173057 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:02:34.173839 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:34.173817 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-75dcfd69c9-dh6qf" Jun 15 06:02:52.235861 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:52.235829 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:02:53.256958 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:02:53.256922 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:06.740307 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.740269 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv"] Jun 15 06:03:06.743244 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.743221 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.745674 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.745653 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\"" Jun 15 06:03:06.752812 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.752788 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv"] Jun 15 06:03:06.773860 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.773820 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:06.839013 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.838977 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.839177 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.839018 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.839177 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.839045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.839255 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.839184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8rx\" (UniqueName: \"kubernetes.io/projected/a14888de-f0a0-4e9d-ba1e-1b826dafa564-kube-api-access-pc8rx\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.839255 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.839215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.839255 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.839247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8rx\" (UniqueName: \"kubernetes.io/projected/a14888de-f0a0-4e9d-ba1e-1b826dafa564-kube-api-access-pc8rx\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940074 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940133 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940180 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940720 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940532 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940720 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940604 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.940720 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.940649 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.943042 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.943015 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.943412 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.943389 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:06.949522 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:06.949502 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8rx\" (UniqueName: \"kubernetes.io/projected/a14888de-f0a0-4e9d-ba1e-1b826dafa564-kube-api-access-pc8rx\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:07.055698 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:07.055659 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:07.188994 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:07.188967 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv"] Jun 15 06:03:07.191051 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:03:07.191022 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14888de_f0a0_4e9d_ba1e_1b826dafa564.slice/crio-61b707510279fbfcf085f3f95085c6bd026540b463101e164133282170084d5c WatchSource:0}: Error finding container 61b707510279fbfcf085f3f95085c6bd026540b463101e164133282170084d5c: Status 404 returned error can't find the container with id 61b707510279fbfcf085f3f95085c6bd026540b463101e164133282170084d5c Jun 15 06:03:07.878555 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:07.878507 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" event={"ID":"a14888de-f0a0-4e9d-ba1e-1b826dafa564","Type":"ContainerStarted","Data":"61b707510279fbfcf085f3f95085c6bd026540b463101e164133282170084d5c"} Jun 15 06:03:08.247605 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.247506 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:08.256535 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.256421 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf"] Jun 15 06:03:08.256808 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.256777 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="main" containerID="cri-o://141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d" gracePeriod=30 Jun 15 06:03:08.256874 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.256815 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="tokenizer" containerID="cri-o://58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa" gracePeriod=30 Jun 15 06:03:08.345941 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.345903 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:08.435856 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.435749 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv"] Jun 15 06:03:08.451980 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.451944 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:08.884604 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.884573 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerID="141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d" exitCode=0 Jun 15 06:03:08.885101 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.884645 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" event={"ID":"a3814454-7fca-44a2-a75f-ca9e86bba7e2","Type":"ContainerDied","Data":"141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d"} Jun 15 06:03:08.886317 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.886287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" event={"ID":"a14888de-f0a0-4e9d-ba1e-1b826dafa564","Type":"ContainerStarted","Data":"494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5"} Jun 15 06:03:08.904215 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:08.904157 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" podStartSLOduration=1.6179144239999999 podStartE2EDuration="2.904136219s" totalImagesPullingTime="1.286221795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:03:08.903321384 +0000 UTC m=+867.248524526" watchObservedRunningTime="2026-06-15 06:03:08.904136219 +0000 UTC m=+867.249339363" Jun 15 06:03:09.605763 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.605734 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:03:09.770480 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770392 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-uds\") pod \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " Jun 15 06:03:09.770480 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770442 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-tmp\") pod \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " Jun 15 06:03:09.770480 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770470 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tls-certs\") pod \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " Jun 15 06:03:09.770783 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770489 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-cache\") pod \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " Jun 15 06:03:09.770783 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770599 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tspf\" (UniqueName: \"kubernetes.io/projected/a3814454-7fca-44a2-a75f-ca9e86bba7e2-kube-api-access-4tspf\") pod \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\" (UID: \"a3814454-7fca-44a2-a75f-ca9e86bba7e2\") " Jun 15 06:03:09.770783 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770732 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "a3814454-7fca-44a2-a75f-ca9e86bba7e2" (UID: "a3814454-7fca-44a2-a75f-ca9e86bba7e2"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:03:09.770783 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770747 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "a3814454-7fca-44a2-a75f-ca9e86bba7e2" (UID: "a3814454-7fca-44a2-a75f-ca9e86bba7e2"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:03:09.770974 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770830 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "a3814454-7fca-44a2-a75f-ca9e86bba7e2" (UID: "a3814454-7fca-44a2-a75f-ca9e86bba7e2"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:03:09.770974 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770866 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:09.770974 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.770879 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:09.772773 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.772752 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3814454-7fca-44a2-a75f-ca9e86bba7e2" (UID: "a3814454-7fca-44a2-a75f-ca9e86bba7e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:03:09.772851 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.772784 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3814454-7fca-44a2-a75f-ca9e86bba7e2-kube-api-access-4tspf" (OuterVolumeSpecName: "kube-api-access-4tspf") pod "a3814454-7fca-44a2-a75f-ca9e86bba7e2" (UID: "a3814454-7fca-44a2-a75f-ca9e86bba7e2"). InnerVolumeSpecName "kube-api-access-4tspf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:03:09.872167 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.872131 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:09.872167 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.872161 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3814454-7fca-44a2-a75f-ca9e86bba7e2-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:09.872167 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.872170 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tspf\" (UniqueName: \"kubernetes.io/projected/a3814454-7fca-44a2-a75f-ca9e86bba7e2-kube-api-access-4tspf\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:09.891996 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.891966 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerID="58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa" exitCode=0 Jun 15 06:03:09.892392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.892013 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" Jun 15 06:03:09.892392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.892027 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" event={"ID":"a3814454-7fca-44a2-a75f-ca9e86bba7e2","Type":"ContainerDied","Data":"58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa"} Jun 15 06:03:09.892392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.892073 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf" event={"ID":"a3814454-7fca-44a2-a75f-ca9e86bba7e2","Type":"ContainerDied","Data":"c4421d51c2d40462749cf03ea5c5f689489bc4f39ad2daf48a52e458cce0064a"} Jun 15 06:03:09.892392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.892095 2569 scope.go:117] "RemoveContainer" containerID="58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa" Jun 15 06:03:09.901055 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.901023 2569 scope.go:117] "RemoveContainer" containerID="141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d" Jun 15 06:03:09.908700 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.908683 2569 scope.go:117] "RemoveContainer" containerID="58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa" Jun 15 06:03:09.908920 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:03:09.908889 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa\": container with ID starting with 58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa not found: ID does not exist" containerID="58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa" Jun 15 06:03:09.908992 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.908919 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa"} err="failed to get container status \"58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa\": rpc error: code = NotFound desc = could not find container \"58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa\": container with ID starting with 58f703feeced4a499e0365dec834c8228c9385195cfbc874ddf4aee5866036fa not found: ID does not exist" Jun 15 06:03:09.908992 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.908935 2569 scope.go:117] "RemoveContainer" containerID="141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d" Jun 15 06:03:09.909195 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:03:09.909177 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d\": container with ID starting with 141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d not found: ID does not exist" containerID="141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d" Jun 15 06:03:09.909238 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.909201 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d"} err="failed to get container status \"141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d\": rpc error: code = NotFound desc = could not find container \"141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d\": container with ID starting with 141f185a264154c81d6fe8d900f2f13b27f4c6abbe8a9b4b864e76036eaf096d not found: ID does not exist" Jun 15 06:03:09.914320 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.914295 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf"] Jun 15 06:03:09.919630 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:09.919609 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5fb5884959rf"] Jun 15 06:03:10.056788 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:10.056758 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" path="/var/lib/kubelet/pods/a3814454-7fca-44a2-a75f-ca9e86bba7e2/volumes" Jun 15 06:03:11.479728 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.479688 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:11.733386 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.733305 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9"] Jun 15 06:03:11.733715 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.733701 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="tokenizer" Jun 15 06:03:11.733774 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.733716 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="tokenizer" Jun 15 06:03:11.733774 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.733727 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="main" Jun 15 06:03:11.733774 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.733733 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="main" Jun 15 06:03:11.733874 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.733798 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="main" Jun 15 06:03:11.733874 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.733807 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3814454-7fca-44a2-a75f-ca9e86bba7e2" containerName="tokenizer" Jun 15 06:03:11.736715 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.736699 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.743960 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.743929 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-dz9xz\"" Jun 15 06:03:11.745195 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.745172 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\"" Jun 15 06:03:11.745449 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.745435 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jun 15 06:03:11.768109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.768071 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9"] Jun 15 06:03:11.770392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.770361 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:11.889232 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.889189 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.889417 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.889292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-tmp\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.889417 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.889322 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.889417 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.889403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-cache\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.889526 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.889433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88n7\" (UniqueName: \"kubernetes.io/projected/9359d817-81c6-4fe2-80e0-76d36607616d-kube-api-access-h88n7\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.889526 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.889460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9359d817-81c6-4fe2-80e0-76d36607616d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990478 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-tmp\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990478 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990422 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990478 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-cache\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990478 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88n7\" (UniqueName: \"kubernetes.io/projected/9359d817-81c6-4fe2-80e0-76d36607616d-kube-api-access-h88n7\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990831 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9359d817-81c6-4fe2-80e0-76d36607616d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990831 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990696 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990946 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990925 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.990984 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990947 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-tmp\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.991022 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.990986 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-tokenizer-cache\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.991060 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.991016 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9359d817-81c6-4fe2-80e0-76d36607616d-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:11.993087 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:11.993065 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9359d817-81c6-4fe2-80e0-76d36607616d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:12.000611 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:12.000581 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88n7\" (UniqueName: \"kubernetes.io/projected/9359d817-81c6-4fe2-80e0-76d36607616d-kube-api-access-h88n7\") pod \"auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9\" (UID: \"9359d817-81c6-4fe2-80e0-76d36607616d\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:12.045948 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:12.045910 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:12.192230 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:12.192205 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9"] Jun 15 06:03:12.194415 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:03:12.194377 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9359d817_81c6_4fe2_80e0_76d36607616d.slice/crio-6596d9759a31ea11f868f452bd18ad155bc2a07dd168f00909997a2e886fc9c3 WatchSource:0}: Error finding container 6596d9759a31ea11f868f452bd18ad155bc2a07dd168f00909997a2e886fc9c3: Status 404 returned error can't find the container with id 6596d9759a31ea11f868f452bd18ad155bc2a07dd168f00909997a2e886fc9c3 Jun 15 06:03:12.909106 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:12.909067 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" event={"ID":"9359d817-81c6-4fe2-80e0-76d36607616d","Type":"ContainerStarted","Data":"55bcaeaed2a108002a4cc5fd3a43c9f6c911f9361281b7d4e5a71122e0d9f91c"} Jun 15 06:03:12.909479 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:12.909127 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" event={"ID":"9359d817-81c6-4fe2-80e0-76d36607616d","Type":"ContainerStarted","Data":"6596d9759a31ea11f868f452bd18ad155bc2a07dd168f00909997a2e886fc9c3"} Jun 15 06:03:13.915114 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:13.915078 2569 generic.go:356] "Generic (PLEG): container finished" podID="9359d817-81c6-4fe2-80e0-76d36607616d" containerID="55bcaeaed2a108002a4cc5fd3a43c9f6c911f9361281b7d4e5a71122e0d9f91c" exitCode=0 Jun 15 06:03:13.915492 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:13.915165 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" event={"ID":"9359d817-81c6-4fe2-80e0-76d36607616d","Type":"ContainerDied","Data":"55bcaeaed2a108002a4cc5fd3a43c9f6c911f9361281b7d4e5a71122e0d9f91c"} Jun 15 06:03:14.922843 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:14.922807 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" event={"ID":"9359d817-81c6-4fe2-80e0-76d36607616d","Type":"ContainerStarted","Data":"d98757a96b0a8e28ff609c63ebd79b7448768eeb80b847f7b2eaf808789a37ce"} Jun 15 06:03:14.923212 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:14.922885 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" event={"ID":"9359d817-81c6-4fe2-80e0-76d36607616d","Type":"ContainerStarted","Data":"f17a74289ac80e1118204b380d94d45d17e4d27afd28e9cb9557783dbe08aeef"} Jun 15 06:03:14.923212 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:14.922919 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:14.945213 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:14.945164 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" podStartSLOduration=3.211794242 podStartE2EDuration="3.945145118s" totalImagesPullingTime="0s" totalInitContainerRuntime="733.350876ms" isStatefulPod=false podCreationTimestamp="2026-06-15 06:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:03:14.944097211 +0000 UTC m=+873.289300371" watchObservedRunningTime="2026-06-15 06:03:14.945145118 +0000 UTC m=+873.290348259" Jun 15 06:03:17.056263 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:17.056232 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:17.075430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:17.075404 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:22.046595 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:22.046562 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:22.047002 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:22.046644 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:22.049602 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:22.049577 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:22.964375 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:22.964340 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:03:23.906480 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:23.906435 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" podUID="a14888de-f0a0-4e9d-ba1e-1b826dafa564" containerName="main" containerID="cri-o://494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5" gracePeriod=45 Jun 15 06:03:24.152952 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.152925 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:24.197120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197042 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-home\") pod \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " Jun 15 06:03:24.197120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197086 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-model-cache\") pod \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " Jun 15 06:03:24.197120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197118 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-dshm\") pod \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " Jun 15 06:03:24.197388 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197136 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pc8rx\" (UniqueName: \"kubernetes.io/projected/a14888de-f0a0-4e9d-ba1e-1b826dafa564-kube-api-access-pc8rx\") pod \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " Jun 15 06:03:24.197388 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197164 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tls-certs\") pod \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " Jun 15 06:03:24.197388 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197179 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tmp-dir\") pod \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\" (UID: \"a14888de-f0a0-4e9d-ba1e-1b826dafa564\") " Jun 15 06:03:24.197388 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197268 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-home" (OuterVolumeSpecName: "home") pod "a14888de-f0a0-4e9d-ba1e-1b826dafa564" (UID: "a14888de-f0a0-4e9d-ba1e-1b826dafa564"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:03:24.197388 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197339 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-model-cache" (OuterVolumeSpecName: "model-cache") pod "a14888de-f0a0-4e9d-ba1e-1b826dafa564" (UID: "a14888de-f0a0-4e9d-ba1e-1b826dafa564"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:03:24.197676 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197506 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a14888de-f0a0-4e9d-ba1e-1b826dafa564" (UID: "a14888de-f0a0-4e9d-ba1e-1b826dafa564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:03:24.197676 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197527 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:24.197676 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.197578 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:24.199359 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.199331 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-dshm" (OuterVolumeSpecName: "dshm") pod "a14888de-f0a0-4e9d-ba1e-1b826dafa564" (UID: "a14888de-f0a0-4e9d-ba1e-1b826dafa564"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:03:24.199359 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.199343 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a14888de-f0a0-4e9d-ba1e-1b826dafa564" (UID: "a14888de-f0a0-4e9d-ba1e-1b826dafa564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:03:24.199465 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.199409 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a14888de-f0a0-4e9d-ba1e-1b826dafa564-kube-api-access-pc8rx" (OuterVolumeSpecName: "kube-api-access-pc8rx") pod "a14888de-f0a0-4e9d-ba1e-1b826dafa564" (UID: "a14888de-f0a0-4e9d-ba1e-1b826dafa564"). InnerVolumeSpecName "kube-api-access-pc8rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:03:24.298755 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.298718 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:24.298755 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.298750 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc8rx\" (UniqueName: \"kubernetes.io/projected/a14888de-f0a0-4e9d-ba1e-1b826dafa564-kube-api-access-pc8rx\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:24.298959 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.298772 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:24.298959 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.298782 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a14888de-f0a0-4e9d-ba1e-1b826dafa564-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:03:24.973430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.973390 2569 generic.go:356] "Generic (PLEG): container finished" podID="a14888de-f0a0-4e9d-ba1e-1b826dafa564" containerID="494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5" exitCode=0 Jun 15 06:03:24.973430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.973441 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" Jun 15 06:03:24.973430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.973456 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" event={"ID":"a14888de-f0a0-4e9d-ba1e-1b826dafa564","Type":"ContainerDied","Data":"494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5"} Jun 15 06:03:24.974081 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.973503 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv" event={"ID":"a14888de-f0a0-4e9d-ba1e-1b826dafa564","Type":"ContainerDied","Data":"61b707510279fbfcf085f3f95085c6bd026540b463101e164133282170084d5c"} Jun 15 06:03:24.974081 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.973524 2569 scope.go:117] "RemoveContainer" containerID="494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5" Jun 15 06:03:24.982182 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.982163 2569 scope.go:117] "RemoveContainer" containerID="494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5" Jun 15 06:03:24.982429 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:03:24.982412 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5\": container with ID starting with 494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5 not found: ID does not exist" containerID="494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5" Jun 15 06:03:24.982497 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:24.982435 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5"} err="failed to get container status \"494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5\": rpc error: code = NotFound desc = could not find container \"494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5\": container with ID starting with 494018c11083b5b2e3d6460a2ecd6eff3fa52cbcd1e7ec70c5c0623ba4af8eb5 not found: ID does not exist" Jun 15 06:03:25.011332 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:25.011293 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv"] Jun 15 06:03:25.017477 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:25.017450 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-bc6865668-glklv"] Jun 15 06:03:26.057289 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:26.057250 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a14888de-f0a0-4e9d-ba1e-1b826dafa564" path="/var/lib/kubelet/pods/a14888de-f0a0-4e9d-ba1e-1b826dafa564/volumes" Jun 15 06:03:31.777843 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:31.777763 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:32.020787 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:32.020745 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:39.491602 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:39.491521 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:03:42.066733 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:42.066701 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:03:42.067662 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:42.067640 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:03:43.969129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:03:43.969083 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9" Jun 15 06:04:06.649970 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:06.649934 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:04:07.614040 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:07.614004 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:04:27.034499 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:27.034462 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:04:36.174902 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.174865 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn"] Jun 15 06:04:36.175267 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.175231 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a14888de-f0a0-4e9d-ba1e-1b826dafa564" containerName="main" Jun 15 06:04:36.175267 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.175243 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a14888de-f0a0-4e9d-ba1e-1b826dafa564" containerName="main" Jun 15 06:04:36.175339 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.175317 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a14888de-f0a0-4e9d-ba1e-1b826dafa564" containerName="main" Jun 15 06:04:36.178351 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.178334 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.180515 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.180495 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\"" Jun 15 06:04:36.187995 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.187968 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn"] Jun 15 06:04:36.189674 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.189646 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:04:36.350273 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.350235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.350467 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.350284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpml\" (UniqueName: \"kubernetes.io/projected/5bd68a63-aca1-43d6-858f-20c701451cc7-kube-api-access-cwpml\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.350467 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.350389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.350467 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.350429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-home\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.350655 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.350503 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-dshm\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.350655 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.350523 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd68a63-aca1-43d6-858f-20c701451cc7-tls-certs\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.350655 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.350579 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-model-cache\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.425449 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.425368 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:04:36.451763 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.451725 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-dshm\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.451763 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.451761 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd68a63-aca1-43d6-858f-20c701451cc7-tls-certs\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452018 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.451787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-model-cache\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452018 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.451910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452018 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.451960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpml\" (UniqueName: \"kubernetes.io/projected/5bd68a63-aca1-43d6-858f-20c701451cc7-kube-api-access-cwpml\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452018 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.452004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452216 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.452058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-home\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452269 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.452240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-model-cache\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452323 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.452302 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452417 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.452393 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.452482 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.452441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-home\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.454028 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.454005 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-dshm\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.454336 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.454314 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd68a63-aca1-43d6-858f-20c701451cc7-tls-certs\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.460517 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.460498 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpml\" (UniqueName: \"kubernetes.io/projected/5bd68a63-aca1-43d6-858f-20c701451cc7-kube-api-access-cwpml\") pod \"precise-prefix-cache-test-kserve-5c556785f6-h6wcn\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.490360 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.490326 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:36.631453 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:36.631428 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn"] Jun 15 06:04:36.633347 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:04:36.633319 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd68a63_aca1_43d6_858f_20c701451cc7.slice/crio-96e998d04eed7173b57c456d164be183cd21a565ef4b0a1c8b40b09ae2eb7e26 WatchSource:0}: Error finding container 96e998d04eed7173b57c456d164be183cd21a565ef4b0a1c8b40b09ae2eb7e26: Status 404 returned error can't find the container with id 96e998d04eed7173b57c456d164be183cd21a565ef4b0a1c8b40b09ae2eb7e26 Jun 15 06:04:37.317990 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:37.317946 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" event={"ID":"5bd68a63-aca1-43d6-858f-20c701451cc7","Type":"ContainerStarted","Data":"0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4"} Jun 15 06:04:37.318462 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:37.318006 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" event={"ID":"5bd68a63-aca1-43d6-858f-20c701451cc7","Type":"ContainerStarted","Data":"96e998d04eed7173b57c456d164be183cd21a565ef4b0a1c8b40b09ae2eb7e26"} Jun 15 06:04:42.347290 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:42.347256 2569 generic.go:356] "Generic (PLEG): container finished" podID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerID="0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4" exitCode=0 Jun 15 06:04:42.347672 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:42.347330 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" event={"ID":"5bd68a63-aca1-43d6-858f-20c701451cc7","Type":"ContainerDied","Data":"0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4"} Jun 15 06:04:43.262609 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:43.262575 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:04:43.354606 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:43.354565 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" event={"ID":"5bd68a63-aca1-43d6-858f-20c701451cc7","Type":"ContainerStarted","Data":"bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb"} Jun 15 06:04:43.375124 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:43.375070 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" podStartSLOduration=1.821843918 podStartE2EDuration="7.375054892s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.553210974s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:04:43.374196263 +0000 UTC m=+961.719399406" watchObservedRunningTime="2026-06-15 06:04:43.375054892 +0000 UTC m=+961.720258033" Jun 15 06:04:46.490474 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:46.490437 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:46.490864 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:46.490499 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:46.502857 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:46.502827 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:04:47.384867 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:04:47.384833 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:05:10.110931 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:10.110833 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn"] Jun 15 06:05:10.114992 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:10.114947 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:05:10.173290 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:10.173245 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:05:25.132966 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.132919 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" podUID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerName="main" containerID="cri-o://bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb" gracePeriod=45 Jun 15 06:05:25.380140 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.380114 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:05:25.399409 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.399347 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd68a63-aca1-43d6-858f-20c701451cc7-tls-certs\") pod \"5bd68a63-aca1-43d6-858f-20c701451cc7\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " Jun 15 06:05:25.399409 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.399387 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-model-cache\") pod \"5bd68a63-aca1-43d6-858f-20c701451cc7\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " Jun 15 06:05:25.400418 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.399439 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-kserve-provision-location\") pod \"5bd68a63-aca1-43d6-858f-20c701451cc7\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " Jun 15 06:05:25.400418 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.399763 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bd68a63-aca1-43d6-858f-20c701451cc7" (UID: "5bd68a63-aca1-43d6-858f-20c701451cc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:05:25.400606 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.400495 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-home\") pod \"5bd68a63-aca1-43d6-858f-20c701451cc7\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " Jun 15 06:05:25.400606 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.400580 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-dshm\") pod \"5bd68a63-aca1-43d6-858f-20c701451cc7\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " Jun 15 06:05:25.400767 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.400639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-tmp-dir\") pod \"5bd68a63-aca1-43d6-858f-20c701451cc7\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " Jun 15 06:05:25.400767 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.400682 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwpml\" (UniqueName: \"kubernetes.io/projected/5bd68a63-aca1-43d6-858f-20c701451cc7-kube-api-access-cwpml\") pod \"5bd68a63-aca1-43d6-858f-20c701451cc7\" (UID: \"5bd68a63-aca1-43d6-858f-20c701451cc7\") " Jun 15 06:05:25.400920 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.400876 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-home" (OuterVolumeSpecName: "home") pod "5bd68a63-aca1-43d6-858f-20c701451cc7" (UID: "5bd68a63-aca1-43d6-858f-20c701451cc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:05:25.401013 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.400972 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:05:25.401013 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.400990 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:05:25.401298 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.401238 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bd68a63-aca1-43d6-858f-20c701451cc7" (UID: "5bd68a63-aca1-43d6-858f-20c701451cc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:05:25.404807 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.404774 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd68a63-aca1-43d6-858f-20c701451cc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bd68a63-aca1-43d6-858f-20c701451cc7" (UID: "5bd68a63-aca1-43d6-858f-20c701451cc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:05:25.405333 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.405294 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd68a63-aca1-43d6-858f-20c701451cc7-kube-api-access-cwpml" (OuterVolumeSpecName: "kube-api-access-cwpml") pod "5bd68a63-aca1-43d6-858f-20c701451cc7" (UID: "5bd68a63-aca1-43d6-858f-20c701451cc7"). InnerVolumeSpecName "kube-api-access-cwpml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:05:25.405433 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.405347 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-dshm" (OuterVolumeSpecName: "dshm") pod "5bd68a63-aca1-43d6-858f-20c701451cc7" (UID: "5bd68a63-aca1-43d6-858f-20c701451cc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:05:25.458708 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.458653 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5bd68a63-aca1-43d6-858f-20c701451cc7" (UID: "5bd68a63-aca1-43d6-858f-20c701451cc7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:05:25.502029 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.501994 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:05:25.502029 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.502027 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpml\" (UniqueName: \"kubernetes.io/projected/5bd68a63-aca1-43d6-858f-20c701451cc7-kube-api-access-cwpml\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:05:25.502029 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.502036 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bd68a63-aca1-43d6-858f-20c701451cc7-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:05:25.502219 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.502044 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:05:25.502219 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.502054 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bd68a63-aca1-43d6-858f-20c701451cc7-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:05:25.555528 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.555494 2569 generic.go:356] "Generic (PLEG): container finished" podID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerID="bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb" exitCode=0 Jun 15 06:05:25.555734 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.555563 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" Jun 15 06:05:25.555734 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.555537 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" event={"ID":"5bd68a63-aca1-43d6-858f-20c701451cc7","Type":"ContainerDied","Data":"bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb"} Jun 15 06:05:25.555734 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.555636 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn" event={"ID":"5bd68a63-aca1-43d6-858f-20c701451cc7","Type":"ContainerDied","Data":"96e998d04eed7173b57c456d164be183cd21a565ef4b0a1c8b40b09ae2eb7e26"} Jun 15 06:05:25.555734 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.555669 2569 scope.go:117] "RemoveContainer" containerID="bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb" Jun 15 06:05:25.564761 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.564738 2569 scope.go:117] "RemoveContainer" containerID="0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4" Jun 15 06:05:25.572707 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.572689 2569 scope.go:117] "RemoveContainer" containerID="bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb" Jun 15 06:05:25.572964 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:05:25.572941 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb\": container with ID starting with bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb not found: ID does not exist" containerID="bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb" Jun 15 06:05:25.573051 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.572977 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb"} err="failed to get container status \"bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb\": rpc error: code = NotFound desc = could not find container \"bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb\": container with ID starting with bdd473a4d9ab91539e6f62e47465f0bc04a949bfb418a352e7854577f50e06cb not found: ID does not exist" Jun 15 06:05:25.573051 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.573013 2569 scope.go:117] "RemoveContainer" containerID="0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4" Jun 15 06:05:25.573288 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:05:25.573270 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4\": container with ID starting with 0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4 not found: ID does not exist" containerID="0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4" Jun 15 06:05:25.573338 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.573295 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4"} err="failed to get container status \"0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4\": rpc error: code = NotFound desc = could not find container \"0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4\": container with ID starting with 0fc653f76c02818e4a5caaa3421b5306274f92ceebb5210c5deabddbbba705e4 not found: ID does not exist" Jun 15 06:05:25.578641 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.578619 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn"] Jun 15 06:05:25.582853 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:25.582832 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5c556785f6-h6wcn"] Jun 15 06:05:26.057237 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:26.057196 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd68a63-aca1-43d6-858f-20c701451cc7" path="/var/lib/kubelet/pods/5bd68a63-aca1-43d6-858f-20c701451cc7/volumes" Jun 15 06:05:26.906048 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:26.906014 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:05:37.677143 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:37.677105 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:05:37.891011 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:37.890975 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:05:44.413771 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:44.413738 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:05:48.889086 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:05:48.889032 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:06:11.070558 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:06:11.070515 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:06:11.128637 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:06:11.128598 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:06:39.182474 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:06:39.182395 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:06:39.448755 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:06:39.448669 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:06:47.219162 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:06:47.219126 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:07:53.267245 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:07:53.267206 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:08:42.095222 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:08:42.095145 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:08:42.098787 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:08:42.098763 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:09:33.540951 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:09:33.540870 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:13:42.131638 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:13:42.131611 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:13:42.135980 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:13:42.135957 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:14:12.157367 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:14:12.157291 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:14:22.782898 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:14:22.782864 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:18:09.260652 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:09.260612 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 06:18:09.270995 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:09.270963 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 06:18:09.281326 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:09.281299 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 06:18:15.969854 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.969814 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94"] Jun 15 06:18:15.970345 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.970182 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerName="storage-initializer" Jun 15 06:18:15.970345 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.970194 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerName="storage-initializer" Jun 15 06:18:15.970345 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.970220 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerName="main" Jun 15 06:18:15.970345 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.970225 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerName="main" Jun 15 06:18:15.970345 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.970282 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd68a63-aca1-43d6-858f-20c701451cc7" containerName="main" Jun 15 06:18:15.973318 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.973301 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:15.975510 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.975492 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\"" Jun 15 06:18:15.993581 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:15.993536 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94"] Jun 15 06:18:16.004814 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.004784 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:18:16.054960 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.054926 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-model-cache\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.055169 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.054972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.055169 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.055051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-home\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.055169 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.055083 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2x6l\" (UniqueName: \"kubernetes.io/projected/270e7d52-45e2-424b-bc89-0d3d866570f5-kube-api-access-t2x6l\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.055169 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.055136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270e7d52-45e2-424b-bc89-0d3d866570f5-tls-certs\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.055384 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.055200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-dshm\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.055384 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.055235 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-tmp-dir\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.155915 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.155877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-tmp-dir\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156094 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.155971 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-model-cache\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156094 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.155997 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156094 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-home\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156265 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2x6l\" (UniqueName: \"kubernetes.io/projected/270e7d52-45e2-424b-bc89-0d3d866570f5-kube-api-access-t2x6l\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156265 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156254 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270e7d52-45e2-424b-bc89-0d3d866570f5-tls-certs\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156368 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156308 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-dshm\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156368 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156341 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-home\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156477 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156477 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156427 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-tmp-dir\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.156477 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.156431 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-model-cache\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.158641 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.158616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-dshm\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.158968 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.158950 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270e7d52-45e2-424b-bc89-0d3d866570f5-tls-certs\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.164643 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.164619 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2x6l\" (UniqueName: \"kubernetes.io/projected/270e7d52-45e2-424b-bc89-0d3d866570f5-kube-api-access-t2x6l\") pod \"auth-invalid-token-test-kserve-f5744d7b7-gjb94\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.218295 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.218257 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b"] Jun 15 06:18:16.222224 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.222159 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.224195 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.224169 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-52pq4\"" Jun 15 06:18:16.232574 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.232531 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b"] Jun 15 06:18:16.283686 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.283652 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:16.357756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.357720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.357892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.357789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-tmp\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.357892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.357816 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-cache\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.357892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.357841 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec3011-3eeb-4339-b447-b91b6325afd3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.358009 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.357915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.358009 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.357945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88n78\" (UniqueName: \"kubernetes.io/projected/6cec3011-3eeb-4339-b447-b91b6325afd3-kube-api-access-88n78\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.431836 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.431796 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94"] Jun 15 06:18:16.435217 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:18:16.435188 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270e7d52_45e2_424b_bc89_0d3d866570f5.slice/crio-859d006cc7eff936b2f73698dc819f290236d612960c9f434d23976fcc856850 WatchSource:0}: Error finding container 859d006cc7eff936b2f73698dc819f290236d612960c9f434d23976fcc856850: Status 404 returned error can't find the container with id 859d006cc7eff936b2f73698dc819f290236d612960c9f434d23976fcc856850 Jun 15 06:18:16.439320 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.439299 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jun 15 06:18:16.458650 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.458627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.458740 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.458666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88n78\" (UniqueName: \"kubernetes.io/projected/6cec3011-3eeb-4339-b447-b91b6325afd3-kube-api-access-88n78\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.458850 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.458826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.458912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.458887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-tmp\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.458976 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.458914 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-cache\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.458976 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.458949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec3011-3eeb-4339-b447-b91b6325afd3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.459152 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.459131 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.459241 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.459219 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-tmp\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.459310 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.459290 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-cache\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.459655 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.459628 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.461634 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.461615 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec3011-3eeb-4339-b447-b91b6325afd3-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.466794 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.466769 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88n78\" (UniqueName: \"kubernetes.io/projected/6cec3011-3eeb-4339-b447-b91b6325afd3-kube-api-access-88n78\") pod \"auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.533061 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.533025 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:16.677595 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:16.677561 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b"] Jun 15 06:18:16.680328 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:18:16.680296 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cec3011_3eeb_4339_b447_b91b6325afd3.slice/crio-c3f2e836d11fd50768b954562a938156cb59f31bdb2923aefa704f2044c9d608 WatchSource:0}: Error finding container c3f2e836d11fd50768b954562a938156cb59f31bdb2923aefa704f2044c9d608: Status 404 returned error can't find the container with id c3f2e836d11fd50768b954562a938156cb59f31bdb2923aefa704f2044c9d608 Jun 15 06:18:17.078697 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:17.078665 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:18:17.181013 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:17.180890 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" event={"ID":"270e7d52-45e2-424b-bc89-0d3d866570f5","Type":"ContainerStarted","Data":"bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a"} Jun 15 06:18:17.181013 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:17.180958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" event={"ID":"270e7d52-45e2-424b-bc89-0d3d866570f5","Type":"ContainerStarted","Data":"859d006cc7eff936b2f73698dc819f290236d612960c9f434d23976fcc856850"} Jun 15 06:18:17.182780 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:17.182747 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerStarted","Data":"48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7"} Jun 15 06:18:17.182930 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:17.182803 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerStarted","Data":"c3f2e836d11fd50768b954562a938156cb59f31bdb2923aefa704f2044c9d608"} Jun 15 06:18:18.189951 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:18.189909 2569 generic.go:356] "Generic (PLEG): container finished" podID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerID="48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7" exitCode=0 Jun 15 06:18:18.190423 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:18.190000 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerDied","Data":"48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7"} Jun 15 06:18:19.197351 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:19.197313 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerStarted","Data":"3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679"} Jun 15 06:18:19.197790 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:19.197382 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerStarted","Data":"2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67"} Jun 15 06:18:19.197790 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:19.197414 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:19.224331 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:19.224257 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" podStartSLOduration=2.4005435520000002 podStartE2EDuration="3.224227346s" totalImagesPullingTime="0s" totalInitContainerRuntime="823.683794ms" isStatefulPod=false podCreationTimestamp="2026-06-15 06:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:18:19.221318707 +0000 UTC m=+1777.566521872" watchObservedRunningTime="2026-06-15 06:18:19.224227346 +0000 UTC m=+1777.569430492" Jun 15 06:18:22.214217 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:22.214179 2569 generic.go:356] "Generic (PLEG): container finished" podID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerID="bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a" exitCode=0 Jun 15 06:18:22.214632 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:22.214238 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" event={"ID":"270e7d52-45e2-424b-bc89-0d3d866570f5","Type":"ContainerDied","Data":"bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a"} Jun 15 06:18:22.215749 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:22.215726 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 15 06:18:25.968844 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:25.968753 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:18:26.533241 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:26.533195 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:26.533451 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:26.533274 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:26.534713 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:26.534677 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.34:8082/healthz\": dial tcp 10.134.0.34:8082: connect: connection refused" Jun 15 06:18:36.535896 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:36.535816 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:36.536999 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:36.536963 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:18:49.362933 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:49.362902 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:18:49.363448 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:49.363173 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:18:50.400057 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:50.400020 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" event={"ID":"270e7d52-45e2-424b-bc89-0d3d866570f5","Type":"ContainerStarted","Data":"6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090"} Jun 15 06:18:50.427598 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:50.427486 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podStartSLOduration=3.373312627 podStartE2EDuration="35.427463724s" totalImagesPullingTime="27.360562803s" totalInitContainerRuntime="4.693588294s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:18:50.425004867 +0000 UTC m=+1808.770208009" watchObservedRunningTime="2026-06-15 06:18:50.427463724 +0000 UTC m=+1808.772666865" Jun 15 06:18:56.284810 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:56.284770 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:56.285209 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:56.284846 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:18:56.286355 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:56.286323 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:18:57.326184 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:18:57.326148 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:19:06.284444 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:19:06.284401 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:19:16.284420 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:19:16.284368 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:19:26.284757 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:19:26.284722 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:19:36.284022 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:19:36.283979 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:19:46.284700 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:19:46.284651 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:19:56.285040 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:19:56.284997 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:20:06.284950 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:06.284843 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:20:16.284083 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:16.284028 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:20:26.284578 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:26.284526 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:20:36.284140 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:36.284088 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.33:8000/health\": dial tcp 10.134.0.33:8000: connect: connection refused" Jun 15 06:20:46.294061 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:46.294032 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:20:46.301825 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:46.301800 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:20:47.547382 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:47.547347 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b"] Jun 15 06:20:47.547818 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:47.547640 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="main" containerID="cri-o://2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67" gracePeriod=30 Jun 15 06:20:47.547818 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:47.547667 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="tokenizer" containerID="cri-o://3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679" gracePeriod=30 Jun 15 06:20:47.556975 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:47.556943 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94"] Jun 15 06:20:47.577513 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:47.577484 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:20:47.662210 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:47.662176 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:20:48.009278 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.009175 2569 generic.go:356] "Generic (PLEG): container finished" podID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerID="2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67" exitCode=0 Jun 15 06:20:48.009278 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.009261 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerDied","Data":"2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67"} Jun 15 06:20:48.806564 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.806523 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:20:48.891312 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891227 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec3011-3eeb-4339-b447-b91b6325afd3-tls-certs\") pod \"6cec3011-3eeb-4339-b447-b91b6325afd3\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " Jun 15 06:20:48.891312 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891266 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-tmp\") pod \"6cec3011-3eeb-4339-b447-b91b6325afd3\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " Jun 15 06:20:48.891573 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891317 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-kserve-provision-location\") pod \"6cec3011-3eeb-4339-b447-b91b6325afd3\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " Jun 15 06:20:48.891573 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891342 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-uds\") pod \"6cec3011-3eeb-4339-b447-b91b6325afd3\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " Jun 15 06:20:48.891573 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891382 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-cache\") pod \"6cec3011-3eeb-4339-b447-b91b6325afd3\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " Jun 15 06:20:48.891573 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891453 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88n78\" (UniqueName: \"kubernetes.io/projected/6cec3011-3eeb-4339-b447-b91b6325afd3-kube-api-access-88n78\") pod \"6cec3011-3eeb-4339-b447-b91b6325afd3\" (UID: \"6cec3011-3eeb-4339-b447-b91b6325afd3\") " Jun 15 06:20:48.891773 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891658 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6cec3011-3eeb-4339-b447-b91b6325afd3" (UID: "6cec3011-3eeb-4339-b447-b91b6325afd3"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:20:48.891773 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891743 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "6cec3011-3eeb-4339-b447-b91b6325afd3" (UID: "6cec3011-3eeb-4339-b447-b91b6325afd3"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:20:48.891773 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891740 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "6cec3011-3eeb-4339-b447-b91b6325afd3" (UID: "6cec3011-3eeb-4339-b447-b91b6325afd3"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:20:48.891895 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.891821 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:20:48.892109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.892089 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6cec3011-3eeb-4339-b447-b91b6325afd3" (UID: "6cec3011-3eeb-4339-b447-b91b6325afd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:20:48.893440 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.893414 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cec3011-3eeb-4339-b447-b91b6325afd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6cec3011-3eeb-4339-b447-b91b6325afd3" (UID: "6cec3011-3eeb-4339-b447-b91b6325afd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:20:48.893578 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.893555 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cec3011-3eeb-4339-b447-b91b6325afd3-kube-api-access-88n78" (OuterVolumeSpecName: "kube-api-access-88n78") pod "6cec3011-3eeb-4339-b447-b91b6325afd3" (UID: "6cec3011-3eeb-4339-b447-b91b6325afd3"). InnerVolumeSpecName "kube-api-access-88n78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:20:48.993120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.993076 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:20:48.993120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.993114 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88n78\" (UniqueName: \"kubernetes.io/projected/6cec3011-3eeb-4339-b447-b91b6325afd3-kube-api-access-88n78\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:20:48.993120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.993130 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6cec3011-3eeb-4339-b447-b91b6325afd3-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:20:48.993362 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.993142 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:20:48.993362 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:48.993154 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6cec3011-3eeb-4339-b447-b91b6325afd3-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:20:49.016788 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.016751 2569 generic.go:356] "Generic (PLEG): container finished" podID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerID="3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679" exitCode=0 Jun 15 06:20:49.016969 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.016816 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" Jun 15 06:20:49.016969 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.016817 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerDied","Data":"3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679"} Jun 15 06:20:49.016969 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.016942 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b" event={"ID":"6cec3011-3eeb-4339-b447-b91b6325afd3","Type":"ContainerDied","Data":"c3f2e836d11fd50768b954562a938156cb59f31bdb2923aefa704f2044c9d608"} Jun 15 06:20:49.016969 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.016965 2569 scope.go:117] "RemoveContainer" containerID="3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679" Jun 15 06:20:49.026307 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.026285 2569 scope.go:117] "RemoveContainer" containerID="2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67" Jun 15 06:20:49.034055 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.034038 2569 scope.go:117] "RemoveContainer" containerID="48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7" Jun 15 06:20:49.039837 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.039815 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b"] Jun 15 06:20:49.042920 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.042900 2569 scope.go:117] "RemoveContainer" containerID="3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679" Jun 15 06:20:49.043228 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:20:49.043207 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679\": container with ID starting with 3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679 not found: ID does not exist" containerID="3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679" Jun 15 06:20:49.043285 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.043236 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679"} err="failed to get container status \"3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679\": rpc error: code = NotFound desc = could not find container \"3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679\": container with ID starting with 3c2b97e9a3dd8688a7e896c2e3cd9b1ef918b6a048778f576589c17fc555c679 not found: ID does not exist" Jun 15 06:20:49.043285 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.043256 2569 scope.go:117] "RemoveContainer" containerID="2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67" Jun 15 06:20:49.043532 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:20:49.043512 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67\": container with ID starting with 2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67 not found: ID does not exist" containerID="2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67" Jun 15 06:20:49.043605 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.043552 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67"} err="failed to get container status \"2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67\": rpc error: code = NotFound desc = could not find container \"2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67\": container with ID starting with 2401cd7f514fb7f0844a3932051eeb3e919fd1c85e5ac04aa9dab12748b7ec67 not found: ID does not exist" Jun 15 06:20:49.043605 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.043568 2569 scope.go:117] "RemoveContainer" containerID="48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7" Jun 15 06:20:49.043831 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:20:49.043806 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7\": container with ID starting with 48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7 not found: ID does not exist" containerID="48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7" Jun 15 06:20:49.043877 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.043842 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7"} err="failed to get container status \"48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7\": rpc error: code = NotFound desc = could not find container \"48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7\": container with ID starting with 48f2a97ceab492ff5f31c4d381ed8d010e4019db58985e24a62181f79940b8b7 not found: ID does not exist" Jun 15 06:20:49.044134 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:49.044116 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-7748b48dbdsc89b"] Jun 15 06:20:50.056406 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:20:50.056372 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" path="/var/lib/kubelet/pods/6cec3011-3eeb-4339-b447-b91b6325afd3/volumes" Jun 15 06:21:03.032726 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.032660 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" containerID="cri-o://6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090" gracePeriod=45 Jun 15 06:21:03.780644 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.780616 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:21:03.934923 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.934832 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2x6l\" (UniqueName: \"kubernetes.io/projected/270e7d52-45e2-424b-bc89-0d3d866570f5-kube-api-access-t2x6l\") pod \"270e7d52-45e2-424b-bc89-0d3d866570f5\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " Jun 15 06:21:03.934923 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.934907 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270e7d52-45e2-424b-bc89-0d3d866570f5-tls-certs\") pod \"270e7d52-45e2-424b-bc89-0d3d866570f5\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " Jun 15 06:21:03.935172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.934949 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-dshm\") pod \"270e7d52-45e2-424b-bc89-0d3d866570f5\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " Jun 15 06:21:03.935172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.934966 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-tmp-dir\") pod \"270e7d52-45e2-424b-bc89-0d3d866570f5\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " Jun 15 06:21:03.935172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.934998 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-kserve-provision-location\") pod \"270e7d52-45e2-424b-bc89-0d3d866570f5\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " Jun 15 06:21:03.935172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.935044 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-home\") pod \"270e7d52-45e2-424b-bc89-0d3d866570f5\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " Jun 15 06:21:03.935172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.935117 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-model-cache\") pod \"270e7d52-45e2-424b-bc89-0d3d866570f5\" (UID: \"270e7d52-45e2-424b-bc89-0d3d866570f5\") " Jun 15 06:21:03.935658 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.935632 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "270e7d52-45e2-424b-bc89-0d3d866570f5" (UID: "270e7d52-45e2-424b-bc89-0d3d866570f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:21:03.935941 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.935905 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-home" (OuterVolumeSpecName: "home") pod "270e7d52-45e2-424b-bc89-0d3d866570f5" (UID: "270e7d52-45e2-424b-bc89-0d3d866570f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:21:03.937216 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.937188 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-dshm" (OuterVolumeSpecName: "dshm") pod "270e7d52-45e2-424b-bc89-0d3d866570f5" (UID: "270e7d52-45e2-424b-bc89-0d3d866570f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:21:03.937440 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.937419 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/270e7d52-45e2-424b-bc89-0d3d866570f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "270e7d52-45e2-424b-bc89-0d3d866570f5" (UID: "270e7d52-45e2-424b-bc89-0d3d866570f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:21:03.937440 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.937428 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270e7d52-45e2-424b-bc89-0d3d866570f5-kube-api-access-t2x6l" (OuterVolumeSpecName: "kube-api-access-t2x6l") pod "270e7d52-45e2-424b-bc89-0d3d866570f5" (UID: "270e7d52-45e2-424b-bc89-0d3d866570f5"). InnerVolumeSpecName "kube-api-access-t2x6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:21:03.946837 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.946808 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "270e7d52-45e2-424b-bc89-0d3d866570f5" (UID: "270e7d52-45e2-424b-bc89-0d3d866570f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:21:03.991042 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:03.990996 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "270e7d52-45e2-424b-bc89-0d3d866570f5" (UID: "270e7d52-45e2-424b-bc89-0d3d866570f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:21:04.035862 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.035835 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:21:04.035862 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.035861 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:21:04.036231 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.035872 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2x6l\" (UniqueName: \"kubernetes.io/projected/270e7d52-45e2-424b-bc89-0d3d866570f5-kube-api-access-t2x6l\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:21:04.036231 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.035881 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/270e7d52-45e2-424b-bc89-0d3d866570f5-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:21:04.036231 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.035889 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:21:04.036231 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.035897 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:21:04.036231 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.035905 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/270e7d52-45e2-424b-bc89-0d3d866570f5-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:21:04.090690 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.090649 2569 generic.go:356] "Generic (PLEG): container finished" podID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerID="6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090" exitCode=0 Jun 15 06:21:04.090856 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.090702 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" Jun 15 06:21:04.090856 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.090707 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" event={"ID":"270e7d52-45e2-424b-bc89-0d3d866570f5","Type":"ContainerDied","Data":"6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090"} Jun 15 06:21:04.090856 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.090766 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94" event={"ID":"270e7d52-45e2-424b-bc89-0d3d866570f5","Type":"ContainerDied","Data":"859d006cc7eff936b2f73698dc819f290236d612960c9f434d23976fcc856850"} Jun 15 06:21:04.090856 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.090790 2569 scope.go:117] "RemoveContainer" containerID="6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090" Jun 15 06:21:04.099376 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.099359 2569 scope.go:117] "RemoveContainer" containerID="bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a" Jun 15 06:21:04.107161 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.107144 2569 scope.go:117] "RemoveContainer" containerID="6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090" Jun 15 06:21:04.107412 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:21:04.107396 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090\": container with ID starting with 6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090 not found: ID does not exist" containerID="6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090" Jun 15 06:21:04.107468 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.107418 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090"} err="failed to get container status \"6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090\": rpc error: code = NotFound desc = could not find container \"6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090\": container with ID starting with 6be2ff20cd0104f15e812b802dcbd20148a59e4032bc650867633bb9e65e5090 not found: ID does not exist" Jun 15 06:21:04.107468 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.107433 2569 scope.go:117] "RemoveContainer" containerID="bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a" Jun 15 06:21:04.107687 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:21:04.107669 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a\": container with ID starting with bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a not found: ID does not exist" containerID="bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a" Jun 15 06:21:04.107734 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.107692 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a"} err="failed to get container status \"bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a\": rpc error: code = NotFound desc = could not find container \"bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a\": container with ID starting with bff4e042f5fdd07b5781830e10b8c957c7f1b9cd8542083a2eb86a10984a178a not found: ID does not exist" Jun 15 06:21:04.110759 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.110735 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94"] Jun 15 06:21:04.116062 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:04.116042 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-f5744d7b7-gjb94"] Jun 15 06:21:06.057122 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:06.057091 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" path="/var/lib/kubelet/pods/270e7d52-45e2-424b-bc89-0d3d866570f5/volumes" Jun 15 06:21:19.167430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167397 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4"] Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167770 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167788 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167803 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="main" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167812 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="main" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167863 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="storage-initializer" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167872 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="storage-initializer" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167911 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="tokenizer" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167919 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="tokenizer" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167935 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="storage-initializer" Jun 15 06:21:19.167971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.167942 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="storage-initializer" Jun 15 06:21:19.168459 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.168039 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="tokenizer" Jun 15 06:21:19.168459 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.168052 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cec3011-3eeb-4339-b447-b91b6325afd3" containerName="main" Jun 15 06:21:19.168459 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.168061 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="270e7d52-45e2-424b-bc89-0d3d866570f5" containerName="main" Jun 15 06:21:19.171513 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.171485 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.173688 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.173665 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\"" Jun 15 06:21:19.184667 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.184640 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4"] Jun 15 06:21:19.193771 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.193737 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:21:19.270852 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.270820 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-tmp-dir\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.270852 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.270855 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-kserve-provision-location\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.271109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.270897 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-dshm\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.271109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.270969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-home\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.271109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.271004 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0e9b79-f547-4bee-8800-e40112b42d3e-tls-certs\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.271109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.271055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4bw\" (UniqueName: \"kubernetes.io/projected/8b0e9b79-f547-4bee-8800-e40112b42d3e-kube-api-access-fh4bw\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.271109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.271093 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-model-cache\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.371715 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.371676 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-tmp-dir\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.371715 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.371716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-kserve-provision-location\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.371976 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.371794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-dshm\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.371976 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.371823 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-home\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.371976 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.371845 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0e9b79-f547-4bee-8800-e40112b42d3e-tls-certs\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.371976 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.371892 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4bw\" (UniqueName: \"kubernetes.io/projected/8b0e9b79-f547-4bee-8800-e40112b42d3e-kube-api-access-fh4bw\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.371976 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.371930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-model-cache\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.372211 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.372043 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-tmp-dir\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.372211 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.372093 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-kserve-provision-location\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.372211 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.372138 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-home\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.372403 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.372384 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-model-cache\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.374155 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.374132 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-dshm\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.374407 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.374388 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0e9b79-f547-4bee-8800-e40112b42d3e-tls-certs\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.380901 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.380878 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4bw\" (UniqueName: \"kubernetes.io/projected/8b0e9b79-f547-4bee-8800-e40112b42d3e-kube-api-access-fh4bw\") pod \"auth-disabled-test-kserve-699694bb49-m6gc4\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.440671 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.440591 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6"] Jun 15 06:21:19.444517 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.444497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.446800 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.446783 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-2p5p8\"" Jun 15 06:21:19.454641 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.454616 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6"] Jun 15 06:21:19.461924 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.461886 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:21:19.472933 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.472892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.473062 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.472956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.473062 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.473035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26km9\" (UniqueName: \"kubernetes.io/projected/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kube-api-access-26km9\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.473062 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.473055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-tmp\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.473184 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.473085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.473184 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.473102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-cache\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.481822 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.481801 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:19.574120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26km9\" (UniqueName: \"kubernetes.io/projected/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kube-api-access-26km9\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574267 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-tmp\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574267 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574267 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-cache\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574267 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574476 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574606 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574583 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-tmp\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574692 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574617 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574692 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574648 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.574836 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.574817 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-cache\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.576907 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.576885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.582861 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.582839 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26km9\" (UniqueName: \"kubernetes.io/projected/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kube-api-access-26km9\") pod \"auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.624671 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.624569 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4"] Jun 15 06:21:19.626835 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:21:19.626803 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0e9b79_f547_4bee_8800_e40112b42d3e.slice/crio-054d7bfda8e18d11eb0441dcd768e65eec58825dda993df9c501e15a9147008d WatchSource:0}: Error finding container 054d7bfda8e18d11eb0441dcd768e65eec58825dda993df9c501e15a9147008d: Status 404 returned error can't find the container with id 054d7bfda8e18d11eb0441dcd768e65eec58825dda993df9c501e15a9147008d Jun 15 06:21:19.756139 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.756040 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:19.900031 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:19.899967 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6"] Jun 15 06:21:19.902055 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:21:19.902020 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55ea47f_ebd0_464d_84a3_952a65e68bb8.slice/crio-0726750574f40ed0dca1a95838403d964320f1aba2c7908964f5a673bdae0155 WatchSource:0}: Error finding container 0726750574f40ed0dca1a95838403d964320f1aba2c7908964f5a673bdae0155: Status 404 returned error can't find the container with id 0726750574f40ed0dca1a95838403d964320f1aba2c7908964f5a673bdae0155 Jun 15 06:21:20.173998 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:20.173957 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerStarted","Data":"9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9"} Jun 15 06:21:20.174512 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:20.174029 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerStarted","Data":"0726750574f40ed0dca1a95838403d964320f1aba2c7908964f5a673bdae0155"} Jun 15 06:21:20.175598 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:20.175571 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" event={"ID":"8b0e9b79-f547-4bee-8800-e40112b42d3e","Type":"ContainerStarted","Data":"fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a"} Jun 15 06:21:20.175777 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:20.175617 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" event={"ID":"8b0e9b79-f547-4bee-8800-e40112b42d3e","Type":"ContainerStarted","Data":"054d7bfda8e18d11eb0441dcd768e65eec58825dda993df9c501e15a9147008d"} Jun 15 06:21:21.182098 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:21.182059 2569 generic.go:356] "Generic (PLEG): container finished" podID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerID="9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9" exitCode=0 Jun 15 06:21:21.182530 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:21.182220 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerDied","Data":"9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9"} Jun 15 06:21:22.189841 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:22.189803 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerStarted","Data":"b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae"} Jun 15 06:21:22.190354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:22.189855 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerStarted","Data":"ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918"} Jun 15 06:21:22.190354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:22.189921 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:22.213413 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:22.213340 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" podStartSLOduration=2.452437346 podStartE2EDuration="3.213318259s" totalImagesPullingTime="0s" totalInitContainerRuntime="760.880913ms" isStatefulPod=false podCreationTimestamp="2026-06-15 06:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:21:22.212051915 +0000 UTC m=+1960.557255069" watchObservedRunningTime="2026-06-15 06:21:22.213318259 +0000 UTC m=+1960.558521403" Jun 15 06:21:27.219320 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:27.219280 2569 generic.go:356] "Generic (PLEG): container finished" podID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerID="fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a" exitCode=0 Jun 15 06:21:27.219701 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:27.219361 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" event={"ID":"8b0e9b79-f547-4bee-8800-e40112b42d3e","Type":"ContainerDied","Data":"fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a"} Jun 15 06:21:28.227215 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:28.227181 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" event={"ID":"8b0e9b79-f547-4bee-8800-e40112b42d3e","Type":"ContainerStarted","Data":"56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422"} Jun 15 06:21:28.249850 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:28.249792 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podStartSLOduration=1.803877978 podStartE2EDuration="9.249771626s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.445893648s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:21:28.246731947 +0000 UTC m=+1966.591935088" watchObservedRunningTime="2026-06-15 06:21:28.249771626 +0000 UTC m=+1966.594974767" Jun 15 06:21:29.482594 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:29.482534 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:29.483010 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:29.482633 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:21:29.484190 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:29.484163 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:21:29.756501 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:29.756398 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:29.756501 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:29.756466 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:29.759230 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:29.759205 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:30.238791 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:30.238756 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:34.736787 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:34.736703 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:21:39.482609 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:39.482562 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:21:49.483027 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:49.482981 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:21:51.245621 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:51.245589 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:21:59.482724 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:21:59.482670 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:22:09.482697 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:09.482638 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:22:19.482281 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:19.482223 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:22:29.482989 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:29.482935 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:22:35.999340 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:35.999306 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:22:36.057593 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:36.057550 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:22:39.482499 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:39.482448 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:22:47.428601 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:47.428533 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:22:49.482752 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:49.482710 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:22:59.482322 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:22:59.482264 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:23:09.483248 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:09.483154 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.35:8000/health\": dial tcp 10.134.0.35:8000: connect: connection refused" Jun 15 06:23:19.492997 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:19.492953 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:23:19.500956 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:19.500928 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:23:24.482302 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:24.482266 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4"] Jun 15 06:23:24.489441 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:24.489407 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6"] Jun 15 06:23:24.489927 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:24.489896 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="main" containerID="cri-o://ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918" gracePeriod=30 Jun 15 06:23:24.490090 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:24.489918 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="tokenizer" containerID="cri-o://b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae" gracePeriod=30 Jun 15 06:23:24.497252 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:24.497204 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:23:24.814432 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:24.814394 2569 generic.go:356] "Generic (PLEG): container finished" podID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerID="ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918" exitCode=0 Jun 15 06:23:24.814637 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:24.814446 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerDied","Data":"ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918"} Jun 15 06:23:25.465225 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.465164 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:23:25.753749 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.753725 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:23:25.821308 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.821274 2569 generic.go:356] "Generic (PLEG): container finished" podID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerID="b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae" exitCode=0 Jun 15 06:23:25.821497 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.821335 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" Jun 15 06:23:25.821497 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.821409 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerDied","Data":"b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae"} Jun 15 06:23:25.821497 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.821466 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6" event={"ID":"f55ea47f-ebd0-464d-84a3-952a65e68bb8","Type":"ContainerDied","Data":"0726750574f40ed0dca1a95838403d964320f1aba2c7908964f5a673bdae0155"} Jun 15 06:23:25.821497 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.821492 2569 scope.go:117] "RemoveContainer" containerID="b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae" Jun 15 06:23:25.830199 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.830181 2569 scope.go:117] "RemoveContainer" containerID="ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918" Jun 15 06:23:25.837823 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.837803 2569 scope.go:117] "RemoveContainer" containerID="9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9" Jun 15 06:23:25.845385 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.845368 2569 scope.go:117] "RemoveContainer" containerID="b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae" Jun 15 06:23:25.845649 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:23:25.845630 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae\": container with ID starting with b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae not found: ID does not exist" containerID="b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae" Jun 15 06:23:25.845700 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.845661 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae"} err="failed to get container status \"b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae\": rpc error: code = NotFound desc = could not find container \"b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae\": container with ID starting with b46cf150b084dc3a1eb3f525242cd0d35fadc9d654d618a0da3fa96d2d3782ae not found: ID does not exist" Jun 15 06:23:25.845700 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.845680 2569 scope.go:117] "RemoveContainer" containerID="ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918" Jun 15 06:23:25.845903 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:23:25.845883 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918\": container with ID starting with ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918 not found: ID does not exist" containerID="ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918" Jun 15 06:23:25.845943 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.845907 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918"} err="failed to get container status \"ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918\": rpc error: code = NotFound desc = could not find container \"ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918\": container with ID starting with ccbd37df6fe3d56b5616d5bffdc7403613cc9f627473ef9d011dde736d904918 not found: ID does not exist" Jun 15 06:23:25.845943 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.845919 2569 scope.go:117] "RemoveContainer" containerID="9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9" Jun 15 06:23:25.846115 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:23:25.846100 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9\": container with ID starting with 9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9 not found: ID does not exist" containerID="9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9" Jun 15 06:23:25.846155 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.846118 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9"} err="failed to get container status \"9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9\": rpc error: code = NotFound desc = could not find container \"9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9\": container with ID starting with 9b9eb51242c198429f26a03a9412124cffafe53fef2f5146e3b484683925dfb9 not found: ID does not exist" Jun 15 06:23:25.895190 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895156 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tls-certs\") pod \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " Jun 15 06:23:25.895376 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895203 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kserve-provision-location\") pod \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " Jun 15 06:23:25.895376 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895250 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-cache\") pod \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " Jun 15 06:23:25.895376 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895340 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26km9\" (UniqueName: \"kubernetes.io/projected/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kube-api-access-26km9\") pod \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " Jun 15 06:23:25.895537 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895412 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-uds\") pod \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " Jun 15 06:23:25.895635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895563 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f55ea47f-ebd0-464d-84a3-952a65e68bb8" (UID: "f55ea47f-ebd0-464d-84a3-952a65e68bb8"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:25.895635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895587 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-tmp\") pod \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\" (UID: \"f55ea47f-ebd0-464d-84a3-952a65e68bb8\") " Jun 15 06:23:25.895752 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895695 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f55ea47f-ebd0-464d-84a3-952a65e68bb8" (UID: "f55ea47f-ebd0-464d-84a3-952a65e68bb8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:25.895982 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.895891 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f55ea47f-ebd0-464d-84a3-952a65e68bb8" (UID: "f55ea47f-ebd0-464d-84a3-952a65e68bb8"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:25.896075 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.896009 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:25.896075 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.896029 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:25.896075 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.896041 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:25.896295 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.896271 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f55ea47f-ebd0-464d-84a3-952a65e68bb8" (UID: "f55ea47f-ebd0-464d-84a3-952a65e68bb8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:25.897917 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.897863 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kube-api-access-26km9" (OuterVolumeSpecName: "kube-api-access-26km9") pod "f55ea47f-ebd0-464d-84a3-952a65e68bb8" (UID: "f55ea47f-ebd0-464d-84a3-952a65e68bb8"). InnerVolumeSpecName "kube-api-access-26km9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:23:25.897917 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.897894 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f55ea47f-ebd0-464d-84a3-952a65e68bb8" (UID: "f55ea47f-ebd0-464d-84a3-952a65e68bb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:23:25.996728 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.996652 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f55ea47f-ebd0-464d-84a3-952a65e68bb8-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:25.996728 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.996680 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:25.996728 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:25.996690 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26km9\" (UniqueName: \"kubernetes.io/projected/f55ea47f-ebd0-464d-84a3-952a65e68bb8-kube-api-access-26km9\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:26.140040 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:26.139994 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6"] Jun 15 06:23:26.145174 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:26.145147 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-b5799d8f5-l8kp6"] Jun 15 06:23:28.057002 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:28.056952 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" path="/var/lib/kubelet/pods/f55ea47f-ebd0-464d-84a3-952a65e68bb8/volumes" Jun 15 06:23:39.514606 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:39.514518 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" containerID="cri-o://56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422" gracePeriod=45 Jun 15 06:23:40.268762 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.268734 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:23:40.431335 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431237 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0e9b79-f547-4bee-8800-e40112b42d3e-tls-certs\") pod \"8b0e9b79-f547-4bee-8800-e40112b42d3e\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " Jun 15 06:23:40.431335 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431310 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-dshm\") pod \"8b0e9b79-f547-4bee-8800-e40112b42d3e\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " Jun 15 06:23:40.431536 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431344 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-tmp-dir\") pod \"8b0e9b79-f547-4bee-8800-e40112b42d3e\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " Jun 15 06:23:40.431536 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431406 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-model-cache\") pod \"8b0e9b79-f547-4bee-8800-e40112b42d3e\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " Jun 15 06:23:40.431536 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431437 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-kserve-provision-location\") pod \"8b0e9b79-f547-4bee-8800-e40112b42d3e\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " Jun 15 06:23:40.431536 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431463 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh4bw\" (UniqueName: \"kubernetes.io/projected/8b0e9b79-f547-4bee-8800-e40112b42d3e-kube-api-access-fh4bw\") pod \"8b0e9b79-f547-4bee-8800-e40112b42d3e\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " Jun 15 06:23:40.431536 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431508 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-home\") pod \"8b0e9b79-f547-4bee-8800-e40112b42d3e\" (UID: \"8b0e9b79-f547-4bee-8800-e40112b42d3e\") " Jun 15 06:23:40.431832 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431757 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b0e9b79-f547-4bee-8800-e40112b42d3e" (UID: "8b0e9b79-f547-4bee-8800-e40112b42d3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:40.432079 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.431919 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:40.432366 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.432341 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-home" (OuterVolumeSpecName: "home") pod "8b0e9b79-f547-4bee-8800-e40112b42d3e" (UID: "8b0e9b79-f547-4bee-8800-e40112b42d3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:40.433534 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.433504 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-dshm" (OuterVolumeSpecName: "dshm") pod "8b0e9b79-f547-4bee-8800-e40112b42d3e" (UID: "8b0e9b79-f547-4bee-8800-e40112b42d3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:40.433917 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.433889 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0e9b79-f547-4bee-8800-e40112b42d3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b0e9b79-f547-4bee-8800-e40112b42d3e" (UID: "8b0e9b79-f547-4bee-8800-e40112b42d3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:23:40.434063 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.434045 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0e9b79-f547-4bee-8800-e40112b42d3e-kube-api-access-fh4bw" (OuterVolumeSpecName: "kube-api-access-fh4bw") pod "8b0e9b79-f547-4bee-8800-e40112b42d3e" (UID: "8b0e9b79-f547-4bee-8800-e40112b42d3e"). InnerVolumeSpecName "kube-api-access-fh4bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:23:40.443964 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.443931 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b0e9b79-f547-4bee-8800-e40112b42d3e" (UID: "8b0e9b79-f547-4bee-8800-e40112b42d3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:40.493442 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.493392 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8b0e9b79-f547-4bee-8800-e40112b42d3e" (UID: "8b0e9b79-f547-4bee-8800-e40112b42d3e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:23:40.533270 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.533231 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:40.533270 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.533268 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:40.533665 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.533280 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:40.533665 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.533290 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4bw\" (UniqueName: \"kubernetes.io/projected/8b0e9b79-f547-4bee-8800-e40112b42d3e-kube-api-access-fh4bw\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:40.533665 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.533298 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0e9b79-f547-4bee-8800-e40112b42d3e-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:40.533665 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.533306 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0e9b79-f547-4bee-8800-e40112b42d3e-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:23:40.898696 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.898660 2569 generic.go:356] "Generic (PLEG): container finished" podID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerID="56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422" exitCode=0 Jun 15 06:23:40.898892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.898705 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" event={"ID":"8b0e9b79-f547-4bee-8800-e40112b42d3e","Type":"ContainerDied","Data":"56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422"} Jun 15 06:23:40.898892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.898721 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" Jun 15 06:23:40.898892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.898763 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4" event={"ID":"8b0e9b79-f547-4bee-8800-e40112b42d3e","Type":"ContainerDied","Data":"054d7bfda8e18d11eb0441dcd768e65eec58825dda993df9c501e15a9147008d"} Jun 15 06:23:40.898892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.898790 2569 scope.go:117] "RemoveContainer" containerID="56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422" Jun 15 06:23:40.907755 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.907737 2569 scope.go:117] "RemoveContainer" containerID="fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a" Jun 15 06:23:40.915838 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.915820 2569 scope.go:117] "RemoveContainer" containerID="56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422" Jun 15 06:23:40.916086 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:23:40.916067 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422\": container with ID starting with 56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422 not found: ID does not exist" containerID="56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422" Jun 15 06:23:40.916132 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.916096 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422"} err="failed to get container status \"56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422\": rpc error: code = NotFound desc = could not find container \"56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422\": container with ID starting with 56189542720aedc4426fccb15313b3bf5e950a419994d49233ecec97f4878422 not found: ID does not exist" Jun 15 06:23:40.916132 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.916115 2569 scope.go:117] "RemoveContainer" containerID="fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a" Jun 15 06:23:40.916363 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:23:40.916345 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a\": container with ID starting with fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a not found: ID does not exist" containerID="fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a" Jun 15 06:23:40.916400 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.916372 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a"} err="failed to get container status \"fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a\": rpc error: code = NotFound desc = could not find container \"fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a\": container with ID starting with fca605a1faddd7c868d96784fdce7a5c4fd68d983402f13c63606a668627e30a not found: ID does not exist" Jun 15 06:23:40.922972 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.922949 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4"] Jun 15 06:23:40.926387 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:40.926363 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-699694bb49-m6gc4"] Jun 15 06:23:42.057093 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:42.057060 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" path="/var/lib/kubelet/pods/8b0e9b79-f547-4bee-8800-e40112b42d3e/volumes" Jun 15 06:23:49.393338 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:49.393305 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:23:49.395766 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:23:49.395413 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:24:07.563007 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:07.562968 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:24:19.598347 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:19.598295 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:24:56.463737 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.463661 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf"] Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464050 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464063 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464083 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="storage-initializer" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464092 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="storage-initializer" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464112 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="storage-initializer" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464120 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="storage-initializer" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464131 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="main" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464139 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="main" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464154 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="tokenizer" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464164 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="tokenizer" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464257 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0e9b79-f547-4bee-8800-e40112b42d3e" containerName="main" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464273 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="tokenizer" Jun 15 06:24:56.464354 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.464284 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f55ea47f-ebd0-464d-84a3-952a65e68bb8" containerName="main" Jun 15 06:24:56.467357 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.467341 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.469829 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.469806 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\"" Jun 15 06:24:56.478107 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.478082 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf"] Jun 15 06:24:56.494299 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.494256 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:24:56.610727 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.610693 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea105c3a-737e-4572-91d7-af35c83e508f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.610727 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.610732 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.610997 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.610835 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.610997 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.610890 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.610997 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.610921 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.610997 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.610945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwnsh\" (UniqueName: \"kubernetes.io/projected/ea105c3a-737e-4572-91d7-af35c83e508f-kube-api-access-xwnsh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.611151 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.611036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.707399 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.707363 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg"] Jun 15 06:24:56.711216 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711192 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.711599 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711533 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.711695 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711654 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.711756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711691 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.711756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711719 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwnsh\" (UniqueName: \"kubernetes.io/projected/ea105c3a-737e-4572-91d7-af35c83e508f-kube-api-access-xwnsh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.711866 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711800 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.711866 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711827 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea105c3a-737e-4572-91d7-af35c83e508f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.711866 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.712030 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.712030 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.711958 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.712500 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.712261 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.712500 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.712309 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.713443 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.713420 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-txszr\"" Jun 15 06:24:56.714493 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.714433 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.714656 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.714631 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea105c3a-737e-4572-91d7-af35c83e508f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.724504 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.724476 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg"] Jun 15 06:24:56.725344 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.725323 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwnsh\" (UniqueName: \"kubernetes.io/projected/ea105c3a-737e-4572-91d7-af35c83e508f-kube-api-access-xwnsh\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.740472 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.740439 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:24:56.778790 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.778752 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:24:56.813646 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.813586 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-tmp\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.813646 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.813644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.813944 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.813707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.813944 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.813764 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.813944 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.813808 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jq2g\" (UniqueName: \"kubernetes.io/projected/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kube-api-access-6jq2g\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.813944 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.813869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-cache\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916382 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.915861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-tmp\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916382 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.915912 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916382 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.915950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916382 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.915994 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916382 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.916041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jq2g\" (UniqueName: \"kubernetes.io/projected/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kube-api-access-6jq2g\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916382 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.916104 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-cache\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916911 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.916674 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-tmp\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916911 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.916738 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-cache\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.916911 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.916833 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.917087 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.916964 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.918239 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.918169 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf"] Jun 15 06:24:56.919158 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.919134 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:56.920619 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:24:56.920592 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea105c3a_737e_4572_91d7_af35c83e508f.slice/crio-3306dd6f7d1a70a15b3e0a1aba696e197e84ec3b58afef2c8b3c9b718d027bbb WatchSource:0}: Error finding container 3306dd6f7d1a70a15b3e0a1aba696e197e84ec3b58afef2c8b3c9b718d027bbb: Status 404 returned error can't find the container with id 3306dd6f7d1a70a15b3e0a1aba696e197e84ec3b58afef2c8b3c9b718d027bbb Jun 15 06:24:56.925016 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:56.924982 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jq2g\" (UniqueName: \"kubernetes.io/projected/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kube-api-access-6jq2g\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:57.041879 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:57.041844 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:57.196502 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:57.196314 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg"] Jun 15 06:24:57.199051 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:24:57.199019 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95631be8_47a4_4b2c_89c8_8e0d64f0407e.slice/crio-fa15b9e4c97fac151d9a982a75d4be5525d4be68c92e65ee17b36b045fcd78eb WatchSource:0}: Error finding container fa15b9e4c97fac151d9a982a75d4be5525d4be68c92e65ee17b36b045fcd78eb: Status 404 returned error can't find the container with id fa15b9e4c97fac151d9a982a75d4be5525d4be68c92e65ee17b36b045fcd78eb Jun 15 06:24:57.278411 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:57.278375 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" event={"ID":"ea105c3a-737e-4572-91d7-af35c83e508f","Type":"ContainerStarted","Data":"3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2"} Jun 15 06:24:57.278628 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:57.278442 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" event={"ID":"ea105c3a-737e-4572-91d7-af35c83e508f","Type":"ContainerStarted","Data":"3306dd6f7d1a70a15b3e0a1aba696e197e84ec3b58afef2c8b3c9b718d027bbb"} Jun 15 06:24:57.280122 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:57.280087 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerStarted","Data":"c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019"} Jun 15 06:24:57.280251 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:57.280143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerStarted","Data":"fa15b9e4c97fac151d9a982a75d4be5525d4be68c92e65ee17b36b045fcd78eb"} Jun 15 06:24:58.286492 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:58.286456 2569 generic.go:356] "Generic (PLEG): container finished" podID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerID="c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019" exitCode=0 Jun 15 06:24:58.286919 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:58.286562 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerDied","Data":"c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019"} Jun 15 06:24:59.294392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:59.294326 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerStarted","Data":"0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446"} Jun 15 06:24:59.294392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:59.294391 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerStarted","Data":"f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72"} Jun 15 06:24:59.294862 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:59.294555 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:24:59.317202 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:24:59.317135 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" podStartSLOduration=2.556651246 podStartE2EDuration="3.317119004s" totalImagesPullingTime="0s" totalInitContainerRuntime="760.467758ms" isStatefulPod=false podCreationTimestamp="2026-06-15 06:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:24:59.314311011 +0000 UTC m=+2177.659514154" watchObservedRunningTime="2026-06-15 06:24:59.317119004 +0000 UTC m=+2177.662322145" Jun 15 06:25:05.054034 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:05.053986 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:25:07.042793 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:07.042755 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:25:07.043432 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:07.042853 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:25:07.047412 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:07.047384 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:25:07.343172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:07.343004 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:25:09.352025 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:09.351993 2569 generic.go:356] "Generic (PLEG): container finished" podID="ea105c3a-737e-4572-91d7-af35c83e508f" containerID="3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2" exitCode=0 Jun 15 06:25:09.352430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:09.352072 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" event={"ID":"ea105c3a-737e-4572-91d7-af35c83e508f","Type":"ContainerDied","Data":"3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2"} Jun 15 06:25:10.358237 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:10.358204 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" event={"ID":"ea105c3a-737e-4572-91d7-af35c83e508f","Type":"ContainerStarted","Data":"a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f"} Jun 15 06:25:10.381359 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:10.381269 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podStartSLOduration=2.7449373059999997 podStartE2EDuration="14.381247541s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.636310235s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:25:10.379169886 +0000 UTC m=+2188.724373045" watchObservedRunningTime="2026-06-15 06:25:10.381247541 +0000 UTC m=+2188.726450684" Jun 15 06:25:16.779740 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:16.779701 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:25:16.780210 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:16.779771 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:25:16.781400 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:16.781363 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:25:26.780080 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:26.780033 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:25:28.347925 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:28.347890 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:25:36.780254 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:36.780199 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:25:41.790757 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:41.790722 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:25:42.028083 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:42.028049 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:25:44.950775 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:44.950737 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:25:46.779973 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:46.779924 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:25:53.694653 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:53.694612 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:25:56.779985 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:25:56.779925 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:26:06.779378 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:26:06.779272 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:26:16.779700 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:26:16.779656 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:26:26.779798 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:26:26.779753 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:26:36.779886 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:26:36.779844 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:26:46.779841 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:26:46.779792 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:26:56.779397 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:26:56.779354 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:27:06.789074 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:06.789040 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:27:06.796560 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:06.796522 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:27:11.568830 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.568597 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg"] Jun 15 06:27:11.569373 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.569061 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="main" containerID="cri-o://f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72" gracePeriod=30 Jun 15 06:27:11.571642 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.569603 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="tokenizer" containerID="cri-o://0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446" gracePeriod=30 Jun 15 06:27:11.572218 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.571793 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:27:11.579418 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.579377 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf"] Jun 15 06:27:11.624498 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.624459 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:27:11.997449 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.997351 2569 generic.go:356] "Generic (PLEG): container finished" podID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerID="f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72" exitCode=0 Jun 15 06:27:11.997449 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:11.997427 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerDied","Data":"f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72"} Jun 15 06:27:12.829469 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.829445 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:27:12.927381 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927281 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-tmp\") pod \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " Jun 15 06:27:12.927381 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927314 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-cache\") pod \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " Jun 15 06:27:12.927666 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927406 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kserve-provision-location\") pod \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " Jun 15 06:27:12.927666 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927467 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-uds\") pod \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " Jun 15 06:27:12.927666 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927506 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jq2g\" (UniqueName: \"kubernetes.io/projected/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kube-api-access-6jq2g\") pod \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " Jun 15 06:27:12.927666 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927627 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "95631be8-47a4-4b2c-89c8-8e0d64f0407e" (UID: "95631be8-47a4-4b2c-89c8-8e0d64f0407e"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:12.927666 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927651 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tls-certs\") pod \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\" (UID: \"95631be8-47a4-4b2c-89c8-8e0d64f0407e\") " Jun 15 06:27:12.927912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927688 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "95631be8-47a4-4b2c-89c8-8e0d64f0407e" (UID: "95631be8-47a4-4b2c-89c8-8e0d64f0407e"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:12.927912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.927707 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "95631be8-47a4-4b2c-89c8-8e0d64f0407e" (UID: "95631be8-47a4-4b2c-89c8-8e0d64f0407e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:12.928028 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.928011 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:12.928081 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.928034 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:12.928081 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.928049 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:12.928081 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.928072 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95631be8-47a4-4b2c-89c8-8e0d64f0407e" (UID: "95631be8-47a4-4b2c-89c8-8e0d64f0407e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:12.929671 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.929643 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "95631be8-47a4-4b2c-89c8-8e0d64f0407e" (UID: "95631be8-47a4-4b2c-89c8-8e0d64f0407e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:27:12.929766 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:12.929715 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kube-api-access-6jq2g" (OuterVolumeSpecName: "kube-api-access-6jq2g") pod "95631be8-47a4-4b2c-89c8-8e0d64f0407e" (UID: "95631be8-47a4-4b2c-89c8-8e0d64f0407e"). InnerVolumeSpecName "kube-api-access-6jq2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:27:13.004122 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.004085 2569 generic.go:356] "Generic (PLEG): container finished" podID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerID="0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446" exitCode=0 Jun 15 06:27:13.004279 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.004147 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" Jun 15 06:27:13.004279 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.004174 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerDied","Data":"0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446"} Jun 15 06:27:13.004279 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.004250 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg" event={"ID":"95631be8-47a4-4b2c-89c8-8e0d64f0407e","Type":"ContainerDied","Data":"fa15b9e4c97fac151d9a982a75d4be5525d4be68c92e65ee17b36b045fcd78eb"} Jun 15 06:27:13.004398 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.004283 2569 scope.go:117] "RemoveContainer" containerID="0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446" Jun 15 06:27:13.014161 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.014141 2569 scope.go:117] "RemoveContainer" containerID="f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72" Jun 15 06:27:13.022385 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.022364 2569 scope.go:117] "RemoveContainer" containerID="c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019" Jun 15 06:27:13.028278 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.028255 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg"] Jun 15 06:27:13.028700 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.028681 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/95631be8-47a4-4b2c-89c8-8e0d64f0407e-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:13.028753 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.028703 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:13.028753 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.028713 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jq2g\" (UniqueName: \"kubernetes.io/projected/95631be8-47a4-4b2c-89c8-8e0d64f0407e-kube-api-access-6jq2g\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:13.030867 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.030847 2569 scope.go:117] "RemoveContainer" containerID="0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446" Jun 15 06:27:13.031118 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:27:13.031100 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446\": container with ID starting with 0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446 not found: ID does not exist" containerID="0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446" Jun 15 06:27:13.031182 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.031127 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446"} err="failed to get container status \"0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446\": rpc error: code = NotFound desc = could not find container \"0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446\": container with ID starting with 0226483e9be4f79bfdb6ff3895c4b035802f83d5aa4d80fcde18c15217db5446 not found: ID does not exist" Jun 15 06:27:13.031182 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.031145 2569 scope.go:117] "RemoveContainer" containerID="f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72" Jun 15 06:27:13.031386 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:27:13.031364 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72\": container with ID starting with f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72 not found: ID does not exist" containerID="f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72" Jun 15 06:27:13.031430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.031394 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72"} err="failed to get container status \"f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72\": rpc error: code = NotFound desc = could not find container \"f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72\": container with ID starting with f972e6972c3251b1105bfac727736cee91498df1bef257c1dbe7206bc0378d72 not found: ID does not exist" Jun 15 06:27:13.031430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.031410 2569 scope.go:117] "RemoveContainer" containerID="c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019" Jun 15 06:27:13.031679 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:27:13.031657 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019\": container with ID starting with c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019 not found: ID does not exist" containerID="c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019" Jun 15 06:27:13.031730 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.031690 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019"} err="failed to get container status \"c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019\": rpc error: code = NotFound desc = could not find container \"c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019\": container with ID starting with c36fd94461e4c3038d64f6e185b349b9a69fbb8e43957bdf99277fa099cac019 not found: ID does not exist" Jun 15 06:27:13.032310 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:13.032292 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schesgxzg"] Jun 15 06:27:14.057261 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:14.057226 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" path="/var/lib/kubelet/pods/95631be8-47a4-4b2c-89c8-8e0d64f0407e/volumes" Jun 15 06:27:26.605750 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:26.605708 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" containerID="cri-o://a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f" gracePeriod=45 Jun 15 06:27:26.789833 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:26.789780 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused" Jun 15 06:27:27.363225 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.363201 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:27:27.468127 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.467975 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwnsh\" (UniqueName: \"kubernetes.io/projected/ea105c3a-737e-4572-91d7-af35c83e508f-kube-api-access-xwnsh\") pod \"ea105c3a-737e-4572-91d7-af35c83e508f\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " Jun 15 06:27:27.468127 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468109 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-dshm\") pod \"ea105c3a-737e-4572-91d7-af35c83e508f\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " Jun 15 06:27:27.468367 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468174 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-model-cache\") pod \"ea105c3a-737e-4572-91d7-af35c83e508f\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " Jun 15 06:27:27.468367 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468202 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-home\") pod \"ea105c3a-737e-4572-91d7-af35c83e508f\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " Jun 15 06:27:27.468367 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468245 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-tmp-dir\") pod \"ea105c3a-737e-4572-91d7-af35c83e508f\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " Jun 15 06:27:27.468367 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468266 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea105c3a-737e-4572-91d7-af35c83e508f-tls-certs\") pod \"ea105c3a-737e-4572-91d7-af35c83e508f\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " Jun 15 06:27:27.468367 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468287 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-kserve-provision-location\") pod \"ea105c3a-737e-4572-91d7-af35c83e508f\" (UID: \"ea105c3a-737e-4572-91d7-af35c83e508f\") " Jun 15 06:27:27.468672 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468396 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea105c3a-737e-4572-91d7-af35c83e508f" (UID: "ea105c3a-737e-4572-91d7-af35c83e508f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:27.468672 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468627 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:27.468982 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.468954 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-home" (OuterVolumeSpecName: "home") pod "ea105c3a-737e-4572-91d7-af35c83e508f" (UID: "ea105c3a-737e-4572-91d7-af35c83e508f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:27.470268 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.470245 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-dshm" (OuterVolumeSpecName: "dshm") pod "ea105c3a-737e-4572-91d7-af35c83e508f" (UID: "ea105c3a-737e-4572-91d7-af35c83e508f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:27.470438 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.470417 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea105c3a-737e-4572-91d7-af35c83e508f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea105c3a-737e-4572-91d7-af35c83e508f" (UID: "ea105c3a-737e-4572-91d7-af35c83e508f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:27:27.470516 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.470449 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea105c3a-737e-4572-91d7-af35c83e508f-kube-api-access-xwnsh" (OuterVolumeSpecName: "kube-api-access-xwnsh") pod "ea105c3a-737e-4572-91d7-af35c83e508f" (UID: "ea105c3a-737e-4572-91d7-af35c83e508f"). InnerVolumeSpecName "kube-api-access-xwnsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:27:27.487029 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.486994 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea105c3a-737e-4572-91d7-af35c83e508f" (UID: "ea105c3a-737e-4572-91d7-af35c83e508f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:27.534948 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.534909 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ea105c3a-737e-4572-91d7-af35c83e508f" (UID: "ea105c3a-737e-4572-91d7-af35c83e508f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:27:27.569053 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.569020 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:27.569053 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.569048 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:27.569053 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.569058 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea105c3a-737e-4572-91d7-af35c83e508f-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:27.569261 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.569067 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:27.569261 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.569078 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwnsh\" (UniqueName: \"kubernetes.io/projected/ea105c3a-737e-4572-91d7-af35c83e508f-kube-api-access-xwnsh\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:27.569261 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:27.569087 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea105c3a-737e-4572-91d7-af35c83e508f-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:27:28.078749 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.078713 2569 generic.go:356] "Generic (PLEG): container finished" podID="ea105c3a-737e-4572-91d7-af35c83e508f" containerID="a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f" exitCode=0 Jun 15 06:27:28.079211 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.078764 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" Jun 15 06:27:28.079211 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.078797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" event={"ID":"ea105c3a-737e-4572-91d7-af35c83e508f","Type":"ContainerDied","Data":"a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f"} Jun 15 06:27:28.079211 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.078849 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf" event={"ID":"ea105c3a-737e-4572-91d7-af35c83e508f","Type":"ContainerDied","Data":"3306dd6f7d1a70a15b3e0a1aba696e197e84ec3b58afef2c8b3c9b718d027bbb"} Jun 15 06:27:28.079211 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.078872 2569 scope.go:117] "RemoveContainer" containerID="a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f" Jun 15 06:27:28.087589 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.087569 2569 scope.go:117] "RemoveContainer" containerID="3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2" Jun 15 06:27:28.095870 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.095843 2569 scope.go:117] "RemoveContainer" containerID="a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f" Jun 15 06:27:28.096108 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:27:28.096090 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f\": container with ID starting with a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f not found: ID does not exist" containerID="a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f" Jun 15 06:27:28.096164 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.096114 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f"} err="failed to get container status \"a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f\": rpc error: code = NotFound desc = could not find container \"a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f\": container with ID starting with a473ab4b427cfe38d7735150a0ed080a6e5c649cce51f82252b726b8fe97885f not found: ID does not exist" Jun 15 06:27:28.096164 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.096130 2569 scope.go:117] "RemoveContainer" containerID="3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2" Jun 15 06:27:28.096356 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:27:28.096336 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2\": container with ID starting with 3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2 not found: ID does not exist" containerID="3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2" Jun 15 06:27:28.096417 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.096366 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2"} err="failed to get container status \"3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2\": rpc error: code = NotFound desc = could not find container \"3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2\": container with ID starting with 3058e5ef6be3d23356025e213b8bf111f04b4c4190b297db492dc99a6f3a0dc2 not found: ID does not exist" Jun 15 06:27:28.100036 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.100016 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf"] Jun 15 06:27:28.104152 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.104131 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6b87f7f5c5mv4vf"] Jun 15 06:27:28.317144 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:28.317062 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:27:30.056817 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:30.056784 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" path="/var/lib/kubelet/pods/ea105c3a-737e-4572-91d7-af35c83e508f/volumes" Jun 15 06:27:34.869694 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.869619 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md"] Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.869988 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="tokenizer" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870002 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="tokenizer" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870012 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="storage-initializer" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870019 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="storage-initializer" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870044 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="main" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870050 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="main" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870057 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="storage-initializer" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870063 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="storage-initializer" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870068 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" Jun 15 06:27:34.870072 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870073 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" Jun 15 06:27:34.870375 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870130 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea105c3a-737e-4572-91d7-af35c83e508f" containerName="main" Jun 15 06:27:34.870375 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870141 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="main" Jun 15 06:27:34.870375 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.870147 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="95631be8-47a4-4b2c-89c8-8e0d64f0407e" containerName="tokenizer" Jun 15 06:27:34.873421 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.873404 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:34.875527 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.875505 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\"" Jun 15 06:27:34.885257 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.885221 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:27:34.886152 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:34.886127 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md"] Jun 15 06:27:35.037352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.037311 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4783d1db-182b-4f02-81f4-b38397c2c11e-tls-certs\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.037589 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.037380 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.037589 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.037427 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-home\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.037589 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.037455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-tmp-dir\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.037589 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.037480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-model-cache\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.037589 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.037512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc4jn\" (UniqueName: \"kubernetes.io/projected/4783d1db-182b-4f02-81f4-b38397c2c11e-kube-api-access-jc4jn\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.037787 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.037610 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-dshm\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.103181 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.103146 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l"] Jun 15 06:27:35.106101 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.106080 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.108415 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.108396 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-r6546\"" Jun 15 06:27:35.118869 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.118841 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l"] Jun 15 06:27:35.132258 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.132188 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:27:35.139041 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139016 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4783d1db-182b-4f02-81f4-b38397c2c11e-tls-certs\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139119 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139119 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139101 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-home\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139202 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-tmp-dir\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139202 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-model-cache\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139202 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139173 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc4jn\" (UniqueName: \"kubernetes.io/projected/4783d1db-182b-4f02-81f4-b38397c2c11e-kube-api-access-jc4jn\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139344 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139212 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-dshm\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139515 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139487 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-home\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139625 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-tmp-dir\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139625 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139605 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.139716 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.139676 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-model-cache\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.141489 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.141465 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-dshm\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.141595 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.141501 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4783d1db-182b-4f02-81f4-b38397c2c11e-tls-certs\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.147661 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.147641 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc4jn\" (UniqueName: \"kubernetes.io/projected/4783d1db-182b-4f02-81f4-b38397c2c11e-kube-api-access-jc4jn\") pod \"custom-route-timeout-test-kserve-598d8c75cc-qw9md\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.183445 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.183413 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:35.240095 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.240062 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.240232 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.240110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.240232 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.240193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.240325 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.240279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3955f416-a9e5-407b-85f3-187e43c54392-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.240376 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.240344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq4ft\" (UniqueName: \"kubernetes.io/projected/3955f416-a9e5-407b-85f3-187e43c54392-kube-api-access-mq4ft\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.240421 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.240373 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.322892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.322857 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md"] Jun 15 06:27:35.325119 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:27:35.325085 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4783d1db_182b_4f02_81f4_b38397c2c11e.slice/crio-8a3c16d3327c3c1ee36daebd1d65dcaeada29f47430a3fbae61c58cf3ca5f99b WatchSource:0}: Error finding container 8a3c16d3327c3c1ee36daebd1d65dcaeada29f47430a3fbae61c58cf3ca5f99b: Status 404 returned error can't find the container with id 8a3c16d3327c3c1ee36daebd1d65dcaeada29f47430a3fbae61c58cf3ca5f99b Jun 15 06:27:35.340953 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.340930 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341063 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.340979 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341063 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341182 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3955f416-a9e5-407b-85f3-187e43c54392-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341269 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq4ft\" (UniqueName: \"kubernetes.io/projected/3955f416-a9e5-407b-85f3-187e43c54392-kube-api-access-mq4ft\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341342 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341342 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341331 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-cache\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341440 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341360 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341440 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341405 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.341605 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.341590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-tmp\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.343759 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.343734 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3955f416-a9e5-407b-85f3-187e43c54392-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.349415 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.349390 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq4ft\" (UniqueName: \"kubernetes.io/projected/3955f416-a9e5-407b-85f3-187e43c54392-kube-api-access-mq4ft\") pod \"custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.416163 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.416079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:35.555292 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:35.555257 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l"] Jun 15 06:27:35.558874 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:27:35.558831 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3955f416_a9e5_407b_85f3_187e43c54392.slice/crio-d492ae6526bf2a2491e004cb25370034e9b6411c8456710232f804b4a1ddfb7b WatchSource:0}: Error finding container d492ae6526bf2a2491e004cb25370034e9b6411c8456710232f804b4a1ddfb7b: Status 404 returned error can't find the container with id d492ae6526bf2a2491e004cb25370034e9b6411c8456710232f804b4a1ddfb7b Jun 15 06:27:36.128601 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:36.128564 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerStarted","Data":"d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924"} Jun 15 06:27:36.129090 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:36.128630 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerStarted","Data":"d492ae6526bf2a2491e004cb25370034e9b6411c8456710232f804b4a1ddfb7b"} Jun 15 06:27:36.130123 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:36.130095 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" event={"ID":"4783d1db-182b-4f02-81f4-b38397c2c11e","Type":"ContainerStarted","Data":"7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67"} Jun 15 06:27:36.130259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:36.130141 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" event={"ID":"4783d1db-182b-4f02-81f4-b38397c2c11e","Type":"ContainerStarted","Data":"8a3c16d3327c3c1ee36daebd1d65dcaeada29f47430a3fbae61c58cf3ca5f99b"} Jun 15 06:27:37.136440 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:37.136403 2569 generic.go:356] "Generic (PLEG): container finished" podID="3955f416-a9e5-407b-85f3-187e43c54392" containerID="d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924" exitCode=0 Jun 15 06:27:37.136869 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:37.136485 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerDied","Data":"d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924"} Jun 15 06:27:38.144701 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:38.144653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerStarted","Data":"5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af"} Jun 15 06:27:38.144701 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:38.144709 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerStarted","Data":"0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26"} Jun 15 06:27:38.145256 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:38.144750 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:38.168651 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:38.168595 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" podStartSLOduration=2.363742775 podStartE2EDuration="3.168577087s" totalImagesPullingTime="0s" totalInitContainerRuntime="804.834312ms" isStatefulPod=false podCreationTimestamp="2026-06-15 06:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:27:38.166726152 +0000 UTC m=+2336.511929296" watchObservedRunningTime="2026-06-15 06:27:38.168577087 +0000 UTC m=+2336.513780228" Jun 15 06:27:42.170044 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:42.169955 2569 generic.go:356] "Generic (PLEG): container finished" podID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerID="7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67" exitCode=0 Jun 15 06:27:42.170044 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:42.169997 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" event={"ID":"4783d1db-182b-4f02-81f4-b38397c2c11e","Type":"ContainerDied","Data":"7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67"} Jun 15 06:27:43.182049 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:43.182002 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" event={"ID":"4783d1db-182b-4f02-81f4-b38397c2c11e","Type":"ContainerStarted","Data":"d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e"} Jun 15 06:27:43.210048 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:43.209976 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podStartSLOduration=2.738334074 podStartE2EDuration="9.209949827s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.471615753s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:27:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:27:43.207467776 +0000 UTC m=+2341.552670921" watchObservedRunningTime="2026-06-15 06:27:43.209949827 +0000 UTC m=+2341.555152969" Jun 15 06:27:44.977017 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:44.976973 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:27:45.183946 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:45.183906 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:45.184126 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:45.183982 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:27:45.185116 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:45.185091 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:27:45.416157 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:45.416120 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:45.416367 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:45.416228 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:45.419156 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:45.419128 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:46.199024 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:46.198994 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:27:55.184811 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:27:55.184759 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:28:05.184287 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:05.184228 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:28:07.203779 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:07.203749 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:28:15.184505 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:15.184458 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:28:25.184254 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:25.184205 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:28:35.184768 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:35.184721 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:28:45.184445 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:45.184403 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:28:49.427892 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:49.427745 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:28:49.431143 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:49.428323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:28:55.184008 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:28:55.183958 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:29:05.184071 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:05.183970 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:29:15.184977 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:15.184922 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:29:25.184081 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:25.184029 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:29:35.194237 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:35.194196 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:29:35.201885 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:35.201859 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:29:39.657258 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:39.657195 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l"] Jun 15 06:29:39.658263 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:39.658199 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="tokenizer" containerID="cri-o://5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af" gracePeriod=30 Jun 15 06:29:39.658414 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:39.658081 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="main" containerID="cri-o://0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26" gracePeriod=30 Jun 15 06:29:39.659358 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:39.659328 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md"] Jun 15 06:29:39.671728 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:39.671697 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:29:39.796169 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:39.796130 2569 generic.go:356] "Generic (PLEG): container finished" podID="3955f416-a9e5-407b-85f3-187e43c54392" containerID="0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26" exitCode=0 Jun 15 06:29:39.796357 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:39.796203 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerDied","Data":"0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26"} Jun 15 06:29:40.799701 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:40.799672 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:29:41.018424 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.018396 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:29:41.143703 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.143664 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-cache\") pod \"3955f416-a9e5-407b-85f3-187e43c54392\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " Jun 15 06:29:41.143703 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.143703 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-uds\") pod \"3955f416-a9e5-407b-85f3-187e43c54392\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " Jun 15 06:29:41.143968 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.143781 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3955f416-a9e5-407b-85f3-187e43c54392-tls-certs\") pod \"3955f416-a9e5-407b-85f3-187e43c54392\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " Jun 15 06:29:41.143968 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.143849 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-tmp\") pod \"3955f416-a9e5-407b-85f3-187e43c54392\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " Jun 15 06:29:41.143968 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.143889 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-kserve-provision-location\") pod \"3955f416-a9e5-407b-85f3-187e43c54392\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " Jun 15 06:29:41.143968 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.143924 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq4ft\" (UniqueName: \"kubernetes.io/projected/3955f416-a9e5-407b-85f3-187e43c54392-kube-api-access-mq4ft\") pod \"3955f416-a9e5-407b-85f3-187e43c54392\" (UID: \"3955f416-a9e5-407b-85f3-187e43c54392\") " Jun 15 06:29:41.144224 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.143974 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "3955f416-a9e5-407b-85f3-187e43c54392" (UID: "3955f416-a9e5-407b-85f3-187e43c54392"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:41.144224 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.144017 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3955f416-a9e5-407b-85f3-187e43c54392" (UID: "3955f416-a9e5-407b-85f3-187e43c54392"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:41.144224 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.144159 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "3955f416-a9e5-407b-85f3-187e43c54392" (UID: "3955f416-a9e5-407b-85f3-187e43c54392"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:41.144392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.144243 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:41.144392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.144256 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:41.144392 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.144265 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:41.144635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.144611 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3955f416-a9e5-407b-85f3-187e43c54392" (UID: "3955f416-a9e5-407b-85f3-187e43c54392"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:41.146088 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.146069 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3955f416-a9e5-407b-85f3-187e43c54392-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3955f416-a9e5-407b-85f3-187e43c54392" (UID: "3955f416-a9e5-407b-85f3-187e43c54392"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:29:41.146146 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.146101 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3955f416-a9e5-407b-85f3-187e43c54392-kube-api-access-mq4ft" (OuterVolumeSpecName: "kube-api-access-mq4ft") pod "3955f416-a9e5-407b-85f3-187e43c54392" (UID: "3955f416-a9e5-407b-85f3-187e43c54392"). InnerVolumeSpecName "kube-api-access-mq4ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:29:41.244745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.244713 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3955f416-a9e5-407b-85f3-187e43c54392-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:41.244745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.244739 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq4ft\" (UniqueName: \"kubernetes.io/projected/3955f416-a9e5-407b-85f3-187e43c54392-kube-api-access-mq4ft\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:41.244745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.244751 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3955f416-a9e5-407b-85f3-187e43c54392-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:41.809008 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.808965 2569 generic.go:356] "Generic (PLEG): container finished" podID="3955f416-a9e5-407b-85f3-187e43c54392" containerID="5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af" exitCode=0 Jun 15 06:29:41.809408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.809037 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" Jun 15 06:29:41.809408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.809048 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerDied","Data":"5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af"} Jun 15 06:29:41.809408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.809114 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l" event={"ID":"3955f416-a9e5-407b-85f3-187e43c54392","Type":"ContainerDied","Data":"d492ae6526bf2a2491e004cb25370034e9b6411c8456710232f804b4a1ddfb7b"} Jun 15 06:29:41.809408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.809149 2569 scope.go:117] "RemoveContainer" containerID="5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af" Jun 15 06:29:41.818791 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.818773 2569 scope.go:117] "RemoveContainer" containerID="0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26" Jun 15 06:29:41.826745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.826727 2569 scope.go:117] "RemoveContainer" containerID="d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924" Jun 15 06:29:41.834235 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.834210 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l"] Jun 15 06:29:41.834779 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.834765 2569 scope.go:117] "RemoveContainer" containerID="5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af" Jun 15 06:29:41.835020 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:29:41.835003 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af\": container with ID starting with 5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af not found: ID does not exist" containerID="5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af" Jun 15 06:29:41.835067 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.835029 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af"} err="failed to get container status \"5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af\": rpc error: code = NotFound desc = could not find container \"5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af\": container with ID starting with 5e5df02a1f55ddcec56cb32503128cc45130a70ebaa83d57b4f8c7b86d60f1af not found: ID does not exist" Jun 15 06:29:41.835112 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.835073 2569 scope.go:117] "RemoveContainer" containerID="0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26" Jun 15 06:29:41.835266 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:29:41.835250 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26\": container with ID starting with 0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26 not found: ID does not exist" containerID="0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26" Jun 15 06:29:41.835307 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.835271 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26"} err="failed to get container status \"0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26\": rpc error: code = NotFound desc = could not find container \"0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26\": container with ID starting with 0f97f7ad402950085f4c0dcb73e2e413f0243d482a46de5a892290c1fcfa2b26 not found: ID does not exist" Jun 15 06:29:41.835307 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.835285 2569 scope.go:117] "RemoveContainer" containerID="d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924" Jun 15 06:29:41.835513 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:29:41.835493 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924\": container with ID starting with d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924 not found: ID does not exist" containerID="d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924" Jun 15 06:29:41.835587 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.835520 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924"} err="failed to get container status \"d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924\": rpc error: code = NotFound desc = could not find container \"d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924\": container with ID starting with d6b876bb3c6f4b71cfea18f37b67a849aa244db3cea7a4324c9af971fe219924 not found: ID does not exist" Jun 15 06:29:41.838436 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:41.838414 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-54bd696fwdw2l"] Jun 15 06:29:42.058837 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:42.058807 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3955f416-a9e5-407b-85f3-187e43c54392" path="/var/lib/kubelet/pods/3955f416-a9e5-407b-85f3-187e43c54392/volumes" Jun 15 06:29:54.692343 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:54.692288 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" containerID="cri-o://d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e" gracePeriod=45 Jun 15 06:29:55.195251 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.195209 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": dial tcp 10.134.0.39:8000: connect: connection refused" Jun 15 06:29:55.442878 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.442849 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:29:55.471977 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.471899 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-tmp-dir\") pod \"4783d1db-182b-4f02-81f4-b38397c2c11e\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " Jun 15 06:29:55.471977 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.471940 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-kserve-provision-location\") pod \"4783d1db-182b-4f02-81f4-b38397c2c11e\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " Jun 15 06:29:55.471977 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.471973 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc4jn\" (UniqueName: \"kubernetes.io/projected/4783d1db-182b-4f02-81f4-b38397c2c11e-kube-api-access-jc4jn\") pod \"4783d1db-182b-4f02-81f4-b38397c2c11e\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " Jun 15 06:29:55.472259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.472002 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-home\") pod \"4783d1db-182b-4f02-81f4-b38397c2c11e\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " Jun 15 06:29:55.472259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.472025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-model-cache\") pod \"4783d1db-182b-4f02-81f4-b38397c2c11e\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " Jun 15 06:29:55.472259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.472110 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4783d1db-182b-4f02-81f4-b38397c2c11e-tls-certs\") pod \"4783d1db-182b-4f02-81f4-b38397c2c11e\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " Jun 15 06:29:55.472259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.472160 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-dshm\") pod \"4783d1db-182b-4f02-81f4-b38397c2c11e\" (UID: \"4783d1db-182b-4f02-81f4-b38397c2c11e\") " Jun 15 06:29:55.472474 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.472438 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-model-cache" (OuterVolumeSpecName: "model-cache") pod "4783d1db-182b-4f02-81f4-b38397c2c11e" (UID: "4783d1db-182b-4f02-81f4-b38397c2c11e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:55.472998 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.472971 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-home" (OuterVolumeSpecName: "home") pod "4783d1db-182b-4f02-81f4-b38397c2c11e" (UID: "4783d1db-182b-4f02-81f4-b38397c2c11e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:55.474332 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.474307 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4783d1db-182b-4f02-81f4-b38397c2c11e-kube-api-access-jc4jn" (OuterVolumeSpecName: "kube-api-access-jc4jn") pod "4783d1db-182b-4f02-81f4-b38397c2c11e" (UID: "4783d1db-182b-4f02-81f4-b38397c2c11e"). InnerVolumeSpecName "kube-api-access-jc4jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:29:55.474785 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.474765 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-dshm" (OuterVolumeSpecName: "dshm") pod "4783d1db-182b-4f02-81f4-b38397c2c11e" (UID: "4783d1db-182b-4f02-81f4-b38397c2c11e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:55.474915 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.474894 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4783d1db-182b-4f02-81f4-b38397c2c11e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4783d1db-182b-4f02-81f4-b38397c2c11e" (UID: "4783d1db-182b-4f02-81f4-b38397c2c11e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:29:55.490806 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.490774 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4783d1db-182b-4f02-81f4-b38397c2c11e" (UID: "4783d1db-182b-4f02-81f4-b38397c2c11e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:55.537144 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.537090 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4783d1db-182b-4f02-81f4-b38397c2c11e" (UID: "4783d1db-182b-4f02-81f4-b38397c2c11e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:29:55.573618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.573580 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4783d1db-182b-4f02-81f4-b38397c2c11e-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:55.573618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.573610 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:55.573618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.573622 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:55.573834 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.573636 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:55.573834 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.573651 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc4jn\" (UniqueName: \"kubernetes.io/projected/4783d1db-182b-4f02-81f4-b38397c2c11e-kube-api-access-jc4jn\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:55.573834 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.573662 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:55.573834 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.573673 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4783d1db-182b-4f02-81f4-b38397c2c11e-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:29:55.873209 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.873173 2569 generic.go:356] "Generic (PLEG): container finished" podID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerID="d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e" exitCode=0 Jun 15 06:29:55.873653 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.873206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" event={"ID":"4783d1db-182b-4f02-81f4-b38397c2c11e","Type":"ContainerDied","Data":"d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e"} Jun 15 06:29:55.873653 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.873233 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" Jun 15 06:29:55.873653 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.873264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md" event={"ID":"4783d1db-182b-4f02-81f4-b38397c2c11e","Type":"ContainerDied","Data":"8a3c16d3327c3c1ee36daebd1d65dcaeada29f47430a3fbae61c58cf3ca5f99b"} Jun 15 06:29:55.873653 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.873288 2569 scope.go:117] "RemoveContainer" containerID="d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e" Jun 15 06:29:55.882662 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.882642 2569 scope.go:117] "RemoveContainer" containerID="7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67" Jun 15 06:29:55.890966 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.890946 2569 scope.go:117] "RemoveContainer" containerID="d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e" Jun 15 06:29:55.891224 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:29:55.891204 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e\": container with ID starting with d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e not found: ID does not exist" containerID="d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e" Jun 15 06:29:55.891273 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.891232 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e"} err="failed to get container status \"d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e\": rpc error: code = NotFound desc = could not find container \"d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e\": container with ID starting with d0994d277d2876974fccf7278fae550c62b612bbaacd8c1037de3e373c5b190e not found: ID does not exist" Jun 15 06:29:55.891273 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.891249 2569 scope.go:117] "RemoveContainer" containerID="7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67" Jun 15 06:29:55.891483 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:29:55.891464 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67\": container with ID starting with 7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67 not found: ID does not exist" containerID="7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67" Jun 15 06:29:55.891520 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.891488 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67"} err="failed to get container status \"7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67\": rpc error: code = NotFound desc = could not find container \"7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67\": container with ID starting with 7881797ae98da87f0ac39de831fb4af3c5d4c2fe53c9f26951fd2b777b1a5e67 not found: ID does not exist" Jun 15 06:29:55.898516 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.898491 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md"] Jun 15 06:29:55.904459 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:55.904435 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-598d8c75cc-qw9md"] Jun 15 06:29:56.056652 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:29:56.056620 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" path="/var/lib/kubelet/pods/4783d1db-182b-4f02-81f4-b38397c2c11e/volumes" Jun 15 06:30:12.160721 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.160685 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx"] Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161094 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="main" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161110 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="main" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161127 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="storage-initializer" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161136 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="storage-initializer" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161151 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="storage-initializer" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161156 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="storage-initializer" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161165 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="tokenizer" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161170 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="tokenizer" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161180 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161185 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161246 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="tokenizer" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161256 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3955f416-a9e5-407b-85f3-187e43c54392" containerName="main" Jun 15 06:30:12.161280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.161262 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4783d1db-182b-4f02-81f4-b38397c2c11e" containerName="main" Jun 15 06:30:12.165818 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.165797 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.168065 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.168043 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\"" Jun 15 06:30:12.175957 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.175933 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx"] Jun 15 06:30:12.188974 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.188948 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:30:12.222126 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.222095 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-home\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.222264 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.222132 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-dshm\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.222264 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.222151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-kserve-provision-location\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.222264 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.222220 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-model-cache\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.222264 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.222246 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-tmp-dir\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.222407 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.222268 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqbj\" (UniqueName: \"kubernetes.io/projected/c15f33fe-f482-409c-8011-41250b69694d-kube-api-access-tdqbj\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.222407 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.222369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15f33fe-f482-409c-8011-41250b69694d-tls-certs\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323353 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15f33fe-f482-409c-8011-41250b69694d-tls-certs\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323353 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-home\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323638 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-dshm\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323638 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-kserve-provision-location\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323638 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-model-cache\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323808 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-tmp-dir\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323808 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323718 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqbj\" (UniqueName: \"kubernetes.io/projected/c15f33fe-f482-409c-8011-41250b69694d-kube-api-access-tdqbj\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323923 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323816 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-home\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.323923 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323867 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-kserve-provision-location\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.324039 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.323940 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-model-cache\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.324095 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.324064 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-tmp-dir\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.325921 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.325901 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c15f33fe-f482-409c-8011-41250b69694d-dshm\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.326053 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.326037 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c15f33fe-f482-409c-8011-41250b69694d-tls-certs\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.331981 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.331957 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqbj\" (UniqueName: \"kubernetes.io/projected/c15f33fe-f482-409c-8011-41250b69694d-kube-api-access-tdqbj\") pod \"router-with-refs-test-kserve-578d595fc-gtvkx\" (UID: \"c15f33fe-f482-409c-8011-41250b69694d\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.477404 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.477304 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:12.617708 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.617681 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx"] Jun 15 06:30:12.619881 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:30:12.619852 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15f33fe_f482_409c_8011_41250b69694d.slice/crio-b6c74c9249f77896656cebbfb73ea23df716890a0ebbbe71f2c040f087d05c6a WatchSource:0}: Error finding container b6c74c9249f77896656cebbfb73ea23df716890a0ebbbe71f2c040f087d05c6a: Status 404 returned error can't find the container with id b6c74c9249f77896656cebbfb73ea23df716890a0ebbbe71f2c040f087d05c6a Jun 15 06:30:12.631083 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.631056 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76"] Jun 15 06:30:12.634802 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.634781 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.636856 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.636833 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-5mggr\"" Jun 15 06:30:12.644139 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.644108 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76"] Jun 15 06:30:12.727018 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.726983 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.727210 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.727025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.727210 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.727067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.727299 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.727208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.727299 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.727250 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rv7r\" (UniqueName: \"kubernetes.io/projected/c837db67-d0c8-447e-87c9-5dec6015f51f-kube-api-access-4rv7r\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.727366 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.727314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837db67-d0c8-447e-87c9-5dec6015f51f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.731685 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.731577 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:30:12.828332 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828332 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828394 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828443 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rv7r\" (UniqueName: \"kubernetes.io/projected/c837db67-d0c8-447e-87c9-5dec6015f51f-kube-api-access-4rv7r\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828516 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837db67-d0c8-447e-87c9-5dec6015f51f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828887 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828796 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-cache\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828887 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828835 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-tmp\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.828990 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.828962 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.829156 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.829139 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c837db67-d0c8-447e-87c9-5dec6015f51f-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.831054 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.831035 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c837db67-d0c8-447e-87c9-5dec6015f51f-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.837285 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.837260 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rv7r\" (UniqueName: \"kubernetes.io/projected/c837db67-d0c8-447e-87c9-5dec6015f51f-kube-api-access-4rv7r\") pod \"router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76\" (UID: \"c837db67-d0c8-447e-87c9-5dec6015f51f\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.955600 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.955563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:12.966759 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.966726 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" event={"ID":"c15f33fe-f482-409c-8011-41250b69694d","Type":"ContainerStarted","Data":"dbadf93e026882e53c6bdf1347713790357cd44393bea4f60a643d328ceaf7e4"} Jun 15 06:30:12.966938 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:12.966784 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" event={"ID":"c15f33fe-f482-409c-8011-41250b69694d","Type":"ContainerStarted","Data":"b6c74c9249f77896656cebbfb73ea23df716890a0ebbbe71f2c040f087d05c6a"} Jun 15 06:30:13.095928 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:13.095901 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76"] Jun 15 06:30:13.098738 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:30:13.098705 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc837db67_d0c8_447e_87c9_5dec6015f51f.slice/crio-79d8120da5120e38d4566dfa283c5d5d63a17a434bc9ef8c6318129dbd46665e WatchSource:0}: Error finding container 79d8120da5120e38d4566dfa283c5d5d63a17a434bc9ef8c6318129dbd46665e: Status 404 returned error can't find the container with id 79d8120da5120e38d4566dfa283c5d5d63a17a434bc9ef8c6318129dbd46665e Jun 15 06:30:13.981058 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:13.981025 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" event={"ID":"c837db67-d0c8-447e-87c9-5dec6015f51f","Type":"ContainerStarted","Data":"baffc1708f3a7efef0c0b049d34c2ca7f8c6aa71d01be02675cd452ffb11b1c6"} Jun 15 06:30:13.981433 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:13.981107 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" event={"ID":"c837db67-d0c8-447e-87c9-5dec6015f51f","Type":"ContainerStarted","Data":"79d8120da5120e38d4566dfa283c5d5d63a17a434bc9ef8c6318129dbd46665e"} Jun 15 06:30:14.987297 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:14.987264 2569 generic.go:356] "Generic (PLEG): container finished" podID="c837db67-d0c8-447e-87c9-5dec6015f51f" containerID="baffc1708f3a7efef0c0b049d34c2ca7f8c6aa71d01be02675cd452ffb11b1c6" exitCode=0 Jun 15 06:30:14.987717 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:14.987329 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" event={"ID":"c837db67-d0c8-447e-87c9-5dec6015f51f","Type":"ContainerDied","Data":"baffc1708f3a7efef0c0b049d34c2ca7f8c6aa71d01be02675cd452ffb11b1c6"} Jun 15 06:30:15.895064 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:15.895022 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:30:15.994045 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:15.994000 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" event={"ID":"c837db67-d0c8-447e-87c9-5dec6015f51f","Type":"ContainerStarted","Data":"f44c996a236d3e1910dee3fb14563f4bc2dc932284021163b6def597a43b3614"} Jun 15 06:30:15.994435 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:15.994072 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:15.994435 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:15.994094 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" event={"ID":"c837db67-d0c8-447e-87c9-5dec6015f51f","Type":"ContainerStarted","Data":"803e4a2f17d5f799d87d17f68a1a177346f5ded01e24303c753de50ba88947ce"} Jun 15 06:30:16.016717 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:16.016656 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" podStartSLOduration=3.198219445 podStartE2EDuration="4.016635904s" totalImagesPullingTime="0s" totalInitContainerRuntime="818.416459ms" isStatefulPod=false podCreationTimestamp="2026-06-15 06:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:30:16.015627405 +0000 UTC m=+2494.360830547" watchObservedRunningTime="2026-06-15 06:30:16.016635904 +0000 UTC m=+2494.361839046" Jun 15 06:30:22.955806 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:22.955760 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:22.956338 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:22.955850 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:22.958815 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:22.958791 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:23.031791 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:23.031763 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:30:24.036911 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:24.036879 2569 generic.go:356] "Generic (PLEG): container finished" podID="c15f33fe-f482-409c-8011-41250b69694d" containerID="dbadf93e026882e53c6bdf1347713790357cd44393bea4f60a643d328ceaf7e4" exitCode=0 Jun 15 06:30:24.037314 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:24.036954 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" event={"ID":"c15f33fe-f482-409c-8011-41250b69694d","Type":"ContainerDied","Data":"dbadf93e026882e53c6bdf1347713790357cd44393bea4f60a643d328ceaf7e4"} Jun 15 06:30:25.044180 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:25.044146 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" event={"ID":"c15f33fe-f482-409c-8011-41250b69694d","Type":"ContainerStarted","Data":"8c7c363c0368b1dd791993f3898677a646e12c2af435e2e198adb3ba9c1f22db"} Jun 15 06:30:25.066238 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:25.066180 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podStartSLOduration=2.309108271 podStartE2EDuration="13.066157524s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.757049253s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:30:25.065242504 +0000 UTC m=+2503.410445647" watchObservedRunningTime="2026-06-15 06:30:25.066157524 +0000 UTC m=+2503.411360665" Jun 15 06:30:32.477872 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:32.477772 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:32.477872 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:32.477844 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:30:32.479575 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:32.479526 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:30:42.478653 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:42.478602 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:30:52.478424 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:52.478372 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:30:54.039439 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:30:54.039395 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76" Jun 15 06:31:02.478010 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:31:02.477965 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:31:11.508813 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:31:11.508758 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:31:12.478002 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:31:12.477952 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:31:22.478289 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:31:22.478239 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:31:32.478038 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:31:32.477998 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:31:42.478725 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:31:42.478683 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:31:52.477943 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:31:52.477884 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:32:02.478074 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:32:02.477979 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:32:12.478086 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:32:12.478036 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" podUID="c15f33fe-f482-409c-8011-41250b69694d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused" Jun 15 06:32:22.487883 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:32:22.487848 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:32:22.495475 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:32:22.495451 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-578d595fc-gtvkx" Jun 15 06:33:49.459803 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:33:49.459691 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:33:49.463677 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:33:49.463653 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:34:48.204877 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:34:48.204830 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:38:49.504606 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:38:49.504468 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:38:49.507476 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:38:49.507456 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:39:39.843007 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:39:39.842913 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:39:39.898438 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:39:39.898387 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:41:19.239480 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:41:19.239393 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:43:49.535428 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:43:49.535395 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:43:49.538836 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:43:49.538808 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:45:05.026338 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:05.026308 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 06:45:05.038730 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:05.038697 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 06:45:05.046858 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:05.046826 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 06:45:05.055592 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:05.055565 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 06:45:05.066959 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:05.066932 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 06:45:13.570120 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.570076 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w"] Jun 15 06:45:13.572765 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.572743 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.575472 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.575447 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\"" Jun 15 06:45:13.575630 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.575471 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Jun 15 06:45:13.595777 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.595743 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w"] Jun 15 06:45:13.595970 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.595949 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.596039 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.595979 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.596104 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.596043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.596160 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.596118 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.596160 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.596141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq8qp\" (UniqueName: \"kubernetes.io/projected/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-kube-api-access-pq8qp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.596265 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.596170 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.596265 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.596201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.596265 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.596247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697126 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697081 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697318 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697318 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq8qp\" (UniqueName: \"kubernetes.io/projected/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-kube-api-access-pq8qp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697318 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697318 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697318 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697266 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697633 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697320 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697633 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697633 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697572 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697793 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697745 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697793 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697772 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.697878 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.697824 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.698171 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.698147 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.699465 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.699441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.699745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.699726 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.706520 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.706493 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq8qp\" (UniqueName: \"kubernetes.io/projected/2bcc7d6e-7945-4c51-910d-4a0ffe93df64-kube-api-access-pq8qp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w\" (UID: \"2bcc7d6e-7945-4c51-910d-4a0ffe93df64\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:13.883610 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:13.883502 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" Jun 15 06:45:14.028917 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:14.028886 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w"] Jun 15 06:45:14.030927 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:45:14.030892 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc7d6e_7945_4c51_910d_4a0ffe93df64.slice/crio-28fb31d88fa49709ed753f6b2208553d12530eb03f6df4e835d2736604231d51 WatchSource:0}: Error finding container 28fb31d88fa49709ed753f6b2208553d12530eb03f6df4e835d2736604231d51: Status 404 returned error can't find the container with id 28fb31d88fa49709ed753f6b2208553d12530eb03f6df4e835d2736604231d51 Jun 15 06:45:14.035604 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:14.035580 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jun 15 06:45:14.303147 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:14.303107 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:45:14.403114 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:14.403081 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:45:14.411966 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:14.411932 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" event={"ID":"2bcc7d6e-7945-4c51-910d-4a0ffe93df64","Type":"ContainerStarted","Data":"47a49ea18aded36d93af1bed5e257738fc523bad3897e0d9cb2050d828fda45d"} Jun 15 06:45:14.412123 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:14.411997 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w" event={"ID":"2bcc7d6e-7945-4c51-910d-4a0ffe93df64","Type":"ContainerStarted","Data":"28fb31d88fa49709ed753f6b2208553d12530eb03f6df4e835d2736604231d51"} Jun 15 06:45:20.939337 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:20.939297 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg"] Jun 15 06:45:20.943966 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:20.943945 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:20.946366 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:20.946345 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-j9bth\"" Jun 15 06:45:20.946484 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:20.946422 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\"" Jun 15 06:45:20.954439 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:20.954418 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg"] Jun 15 06:45:21.044325 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.044280 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467"] Jun 15 06:45:21.048197 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.048172 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.059784 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.059757 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467"] Jun 15 06:45:21.069517 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.069485 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.069676 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.069533 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.069676 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.069655 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.069776 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.069694 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzpxr\" (UniqueName: \"kubernetes.io/projected/fdf25b94-5830-4064-89ad-ac02b4a46325-kube-api-access-tzpxr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.069822 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.069780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.069822 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.069814 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.069895 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.069839 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf25b94-5830-4064-89ad-ac02b4a46325-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.133722 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.133684 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:45:21.170365 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170328 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.170365 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170367 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jpmc\" (UniqueName: \"kubernetes.io/projected/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kube-api-access-6jpmc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170407 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170434 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170455 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf25b94-5830-4064-89ad-ac02b4a46325-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170506 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170583 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.170618 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.171032 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.171032 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.171032 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.171032 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzpxr\" (UniqueName: \"kubernetes.io/projected/fdf25b94-5830-4064-89ad-ac02b4a46325-kube-api-access-tzpxr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.171032 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170915 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.171032 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.170915 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.171271 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.171059 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.171271 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.171122 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.172811 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.172792 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.172957 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.172941 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf25b94-5830-4064-89ad-ac02b4a46325-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.179937 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.179909 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzpxr\" (UniqueName: \"kubernetes.io/projected/fdf25b94-5830-4064-89ad-ac02b4a46325-kube-api-access-tzpxr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.255672 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.255585 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:21.271894 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.271856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.272041 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.271916 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.272041 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.271988 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.272041 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.272022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jpmc\" (UniqueName: \"kubernetes.io/projected/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kube-api-access-6jpmc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.272200 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.272084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.272200 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.272143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.272200 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.272175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.272361 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.272275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.273867 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.272724 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.273867 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.272786 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.273867 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.273320 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.274803 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.274776 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.275459 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.275441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.281110 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.281082 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jpmc\" (UniqueName: \"kubernetes.io/projected/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kube-api-access-6jpmc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.359971 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.359938 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:21.402868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.402835 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg"] Jun 15 06:45:21.404206 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:45:21.404179 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf25b94_5830_4064_89ad_ac02b4a46325.slice/crio-4bf65fdeceb059f8953c2bafd096a817979b64c3ea0c8edfa5b345d418acdd79 WatchSource:0}: Error finding container 4bf65fdeceb059f8953c2bafd096a817979b64c3ea0c8edfa5b345d418acdd79: Status 404 returned error can't find the container with id 4bf65fdeceb059f8953c2bafd096a817979b64c3ea0c8edfa5b345d418acdd79 Jun 15 06:45:21.406832 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.406804 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jun 15 06:45:21.448425 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.448375 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerStarted","Data":"4bf65fdeceb059f8953c2bafd096a817979b64c3ea0c8edfa5b345d418acdd79"} Jun 15 06:45:21.508096 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.508069 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467"] Jun 15 06:45:21.509935 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:45:21.509902 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4babe17d_2700_4b97_a9f4_d5fc4bb7045f.slice/crio-eae82ea4995755efe3d9305b0f8fdfb05926225e475d6f060595bd3b7276997e WatchSource:0}: Error finding container eae82ea4995755efe3d9305b0f8fdfb05926225e475d6f060595bd3b7276997e: Status 404 returned error can't find the container with id eae82ea4995755efe3d9305b0f8fdfb05926225e475d6f060595bd3b7276997e Jun 15 06:45:21.936666 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:21.936626 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:45:22.454923 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:22.454884 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" event={"ID":"4babe17d-2700-4b97-a9f4-d5fc4bb7045f","Type":"ContainerStarted","Data":"3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd"} Jun 15 06:45:22.455350 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:22.454949 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" event={"ID":"4babe17d-2700-4b97-a9f4-d5fc4bb7045f","Type":"ContainerStarted","Data":"eae82ea4995755efe3d9305b0f8fdfb05926225e475d6f060595bd3b7276997e"} Jun 15 06:45:22.456430 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:22.456404 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerStarted","Data":"34d3223157f99e605587e958adce6e67e80bc34be5bb2385656af21d7bd778e4"} Jun 15 06:45:22.456529 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:22.456462 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:23.462844 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:23.462803 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerStarted","Data":"f49e411f65fc8911b341e11db05c5f894c18fc22e2e1fe70fa23dbaaa0333b3f"} Jun 15 06:45:32.511205 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:32.511170 2569 generic.go:356] "Generic (PLEG): container finished" podID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerID="f49e411f65fc8911b341e11db05c5f894c18fc22e2e1fe70fa23dbaaa0333b3f" exitCode=0 Jun 15 06:45:32.511759 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:32.511251 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerDied","Data":"f49e411f65fc8911b341e11db05c5f894c18fc22e2e1fe70fa23dbaaa0333b3f"} Jun 15 06:45:32.513296 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:32.513268 2569 generic.go:356] "Generic (PLEG): container finished" podID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerID="3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd" exitCode=0 Jun 15 06:45:32.513433 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:32.513344 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" event={"ID":"4babe17d-2700-4b97-a9f4-d5fc4bb7045f","Type":"ContainerDied","Data":"3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd"} Jun 15 06:45:32.527073 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:32.527007 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:33.523046 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:33.522997 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" event={"ID":"4babe17d-2700-4b97-a9f4-d5fc4bb7045f","Type":"ContainerStarted","Data":"d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896"} Jun 15 06:45:33.529217 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:33.529125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerStarted","Data":"a32561964cba0f3d0afebc149d764b236472d97bce39d0faddac1898efe87715"} Jun 15 06:45:33.551736 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:33.551668 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podStartSLOduration=2.44041149 podStartE2EDuration="12.55163617s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.11122468s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:45:33.551240183 +0000 UTC m=+3411.896443360" watchObservedRunningTime="2026-06-15 06:45:33.55163617 +0000 UTC m=+3411.896839312" Jun 15 06:45:33.576692 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:33.576630 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podStartSLOduration=3.445355381 podStartE2EDuration="13.576612666s" totalImagesPullingTime="867.72316ms" totalInitContainerRuntime="9.263534125s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:45:33.57456449 +0000 UTC m=+3411.919767633" watchObservedRunningTime="2026-06-15 06:45:33.576612666 +0000 UTC m=+3411.921815806" Jun 15 06:45:38.270926 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:38.270889 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:45:41.255985 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:41.255940 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:41.257022 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:41.256953 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:45:41.257951 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:41.257654 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:45:41.361146 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:41.361085 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:41.361355 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:41.361199 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:45:41.363094 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:41.363054 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:45:48.341137 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:48.341095 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:45:51.256944 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:51.256893 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:45:51.361200 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:45:51.361148 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:46:01.256495 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:01.256438 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:46:01.361303 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:01.361254 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:46:11.257189 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:11.256905 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:46:11.360736 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:11.360688 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:46:21.256584 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:21.256314 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:46:21.361400 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:21.361351 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:46:31.256126 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:31.256076 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:46:31.360808 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:31.360764 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:46:41.256336 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:41.256279 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:46:41.360704 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:41.360658 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:46:51.256719 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:51.256677 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:46:51.361357 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:46:51.361299 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:47:01.256175 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:01.256077 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:47:01.360900 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:01.360852 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:47:11.257366 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:11.257291 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:47:11.361498 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:11.361443 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:47:21.257004 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:21.256959 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:47:21.361129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:21.361080 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:47:31.256062 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:31.256013 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:47:31.360363 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:31.360314 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:47:41.256219 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:41.256179 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:47:41.361032 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:41.360986 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:47:51.256434 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:51.256389 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:47:51.361196 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:47:51.361150 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:48:01.256993 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:01.256946 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:48:01.360791 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:01.360741 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:48:11.256889 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:11.256840 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused" Jun 15 06:48:11.361235 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:11.361180 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused" Jun 15 06:48:21.270305 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:21.270261 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:48:21.282389 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:21.282361 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:48:21.370602 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:21.370537 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:48:21.380640 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:21.380606 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:48:27.490983 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:27.490936 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg"] Jun 15 06:48:27.494372 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:27.494340 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467"] Jun 15 06:48:27.528677 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:27.528640 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:48:28.397226 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:28.397190 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:48:42.518172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:42.518108 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" containerID="cri-o://a32561964cba0f3d0afebc149d764b236472d97bce39d0faddac1898efe87715" gracePeriod=45 Jun 15 06:48:42.521624 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:42.521585 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" containerID="cri-o://d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896" gracePeriod=45 Jun 15 06:48:43.436816 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.436772 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="llm-d-routing-sidecar" containerID="cri-o://34d3223157f99e605587e958adce6e67e80bc34be5bb2385656af21d7bd778e4" gracePeriod=45 Jun 15 06:48:43.573329 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.573295 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:48:43.624021 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.623988 2569 generic.go:356] "Generic (PLEG): container finished" podID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerID="a32561964cba0f3d0afebc149d764b236472d97bce39d0faddac1898efe87715" exitCode=0 Jun 15 06:48:43.624021 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.624016 2569 generic.go:356] "Generic (PLEG): container finished" podID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerID="34d3223157f99e605587e958adce6e67e80bc34be5bb2385656af21d7bd778e4" exitCode=0 Jun 15 06:48:43.624255 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.624065 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerDied","Data":"a32561964cba0f3d0afebc149d764b236472d97bce39d0faddac1898efe87715"} Jun 15 06:48:43.624255 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.624133 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerDied","Data":"34d3223157f99e605587e958adce6e67e80bc34be5bb2385656af21d7bd778e4"} Jun 15 06:48:43.625906 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.625881 2569 generic.go:356] "Generic (PLEG): container finished" podID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerID="d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896" exitCode=0 Jun 15 06:48:43.626014 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.625936 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" event={"ID":"4babe17d-2700-4b97-a9f4-d5fc4bb7045f","Type":"ContainerDied","Data":"d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896"} Jun 15 06:48:43.626014 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.625946 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" Jun 15 06:48:43.626014 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.625984 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467" event={"ID":"4babe17d-2700-4b97-a9f4-d5fc4bb7045f","Type":"ContainerDied","Data":"eae82ea4995755efe3d9305b0f8fdfb05926225e475d6f060595bd3b7276997e"} Jun 15 06:48:43.626132 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.626015 2569 scope.go:117] "RemoveContainer" containerID="d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896" Jun 15 06:48:43.636199 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.636162 2569 scope.go:117] "RemoveContainer" containerID="3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd" Jun 15 06:48:43.646252 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.646219 2569 scope.go:117] "RemoveContainer" containerID="d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896" Jun 15 06:48:43.646742 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:48:43.646714 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896\": container with ID starting with d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896 not found: ID does not exist" containerID="d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896" Jun 15 06:48:43.646857 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.646753 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896"} err="failed to get container status \"d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896\": rpc error: code = NotFound desc = could not find container \"d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896\": container with ID starting with d9fb265d06e37e7fae44ceebd3ab909db8e8e206b4b542d77b636e5e2cd6b896 not found: ID does not exist" Jun 15 06:48:43.646857 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.646779 2569 scope.go:117] "RemoveContainer" containerID="3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd" Jun 15 06:48:43.647100 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:48:43.647073 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd\": container with ID starting with 3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd not found: ID does not exist" containerID="3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd" Jun 15 06:48:43.647172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.647109 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd"} err="failed to get container status \"3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd\": rpc error: code = NotFound desc = could not find container \"3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd\": container with ID starting with 3323407257e80c22abd401eaaa6eeaebea8bc053e3b9992005b55fa9e1639dbd not found: ID does not exist" Jun 15 06:48:43.691369 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.691341 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kserve-provision-location\") pod \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " Jun 15 06:48:43.691563 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.691389 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tmp-dir\") pod \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " Jun 15 06:48:43.691563 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.691421 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-home\") pod \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " Jun 15 06:48:43.691563 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.691461 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jpmc\" (UniqueName: \"kubernetes.io/projected/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kube-api-access-6jpmc\") pod \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " Jun 15 06:48:43.691563 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.691485 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tls-certs\") pod \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " Jun 15 06:48:43.691563 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.691521 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-dshm\") pod \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " Jun 15 06:48:43.691844 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.691632 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-model-cache\") pod \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\" (UID: \"4babe17d-2700-4b97-a9f4-d5fc4bb7045f\") " Jun 15 06:48:43.692289 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.692248 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-home" (OuterVolumeSpecName: "home") pod "4babe17d-2700-4b97-a9f4-d5fc4bb7045f" (UID: "4babe17d-2700-4b97-a9f4-d5fc4bb7045f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.692289 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.692271 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4babe17d-2700-4b97-a9f4-d5fc4bb7045f" (UID: "4babe17d-2700-4b97-a9f4-d5fc4bb7045f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.692794 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.692768 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:48:43.694092 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.694063 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4babe17d-2700-4b97-a9f4-d5fc4bb7045f" (UID: "4babe17d-2700-4b97-a9f4-d5fc4bb7045f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:48:43.694247 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.694216 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kube-api-access-6jpmc" (OuterVolumeSpecName: "kube-api-access-6jpmc") pod "4babe17d-2700-4b97-a9f4-d5fc4bb7045f" (UID: "4babe17d-2700-4b97-a9f4-d5fc4bb7045f"). InnerVolumeSpecName "kube-api-access-6jpmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:48:43.694335 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.694252 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-dshm" (OuterVolumeSpecName: "dshm") pod "4babe17d-2700-4b97-a9f4-d5fc4bb7045f" (UID: "4babe17d-2700-4b97-a9f4-d5fc4bb7045f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.710693 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.710658 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4babe17d-2700-4b97-a9f4-d5fc4bb7045f" (UID: "4babe17d-2700-4b97-a9f4-d5fc4bb7045f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.738893 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.738851 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4babe17d-2700-4b97-a9f4-d5fc4bb7045f" (UID: "4babe17d-2700-4b97-a9f4-d5fc4bb7045f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.792778 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.792744 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf25b94-5830-4064-89ad-ac02b4a46325-tls-certs\") pod \"fdf25b94-5830-4064-89ad-ac02b4a46325\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " Jun 15 06:48:43.793019 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.792786 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-dshm\") pod \"fdf25b94-5830-4064-89ad-ac02b4a46325\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " Jun 15 06:48:43.793019 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.792821 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzpxr\" (UniqueName: \"kubernetes.io/projected/fdf25b94-5830-4064-89ad-ac02b4a46325-kube-api-access-tzpxr\") pod \"fdf25b94-5830-4064-89ad-ac02b4a46325\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " Jun 15 06:48:43.793019 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.792846 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-home\") pod \"fdf25b94-5830-4064-89ad-ac02b4a46325\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " Jun 15 06:48:43.793019 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.792862 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-kserve-provision-location\") pod \"fdf25b94-5830-4064-89ad-ac02b4a46325\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " Jun 15 06:48:43.793019 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.792892 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-model-cache\") pod \"fdf25b94-5830-4064-89ad-ac02b4a46325\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " Jun 15 06:48:43.793019 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.792945 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-tmp-dir\") pod \"fdf25b94-5830-4064-89ad-ac02b4a46325\" (UID: \"fdf25b94-5830-4064-89ad-ac02b4a46325\") " Jun 15 06:48:43.793352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793220 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdf25b94-5830-4064-89ad-ac02b4a46325" (UID: "fdf25b94-5830-4064-89ad-ac02b4a46325"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.793352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793285 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jpmc\" (UniqueName: \"kubernetes.io/projected/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kube-api-access-6jpmc\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.793352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793303 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.793352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793315 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.793352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793327 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.793352 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793340 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.793671 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793355 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.793671 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793369 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4babe17d-2700-4b97-a9f4-d5fc4bb7045f-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.793671 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.793650 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-home" (OuterVolumeSpecName: "home") pod "fdf25b94-5830-4064-89ad-ac02b4a46325" (UID: "fdf25b94-5830-4064-89ad-ac02b4a46325"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.795058 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.795029 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdf25b94-5830-4064-89ad-ac02b4a46325-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdf25b94-5830-4064-89ad-ac02b4a46325" (UID: "fdf25b94-5830-4064-89ad-ac02b4a46325"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:48:43.795190 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.795168 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdf25b94-5830-4064-89ad-ac02b4a46325-kube-api-access-tzpxr" (OuterVolumeSpecName: "kube-api-access-tzpxr") pod "fdf25b94-5830-4064-89ad-ac02b4a46325" (UID: "fdf25b94-5830-4064-89ad-ac02b4a46325"). InnerVolumeSpecName "kube-api-access-tzpxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:48:43.795380 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.795367 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-dshm" (OuterVolumeSpecName: "dshm") pod "fdf25b94-5830-4064-89ad-ac02b4a46325" (UID: "fdf25b94-5830-4064-89ad-ac02b4a46325"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.804959 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.804899 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdf25b94-5830-4064-89ad-ac02b4a46325" (UID: "fdf25b94-5830-4064-89ad-ac02b4a46325"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.811139 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.811110 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fdf25b94-5830-4064-89ad-ac02b4a46325" (UID: "fdf25b94-5830-4064-89ad-ac02b4a46325"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:48:43.894006 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.893966 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.894006 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.894001 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzpxr\" (UniqueName: \"kubernetes.io/projected/fdf25b94-5830-4064-89ad-ac02b4a46325-kube-api-access-tzpxr\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.894006 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.894012 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.894259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.894021 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.894259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.894029 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.894259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.894039 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdf25b94-5830-4064-89ad-ac02b4a46325-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.894259 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.894046 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdf25b94-5830-4064-89ad-ac02b4a46325-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:48:43.952514 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.952479 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467"] Jun 15 06:48:43.959172 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:43.959142 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-864m467"] Jun 15 06:48:44.057655 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.057571 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" path="/var/lib/kubelet/pods/4babe17d-2700-4b97-a9f4-d5fc4bb7045f/volumes" Jun 15 06:48:44.634114 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.634079 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" event={"ID":"fdf25b94-5830-4064-89ad-ac02b4a46325","Type":"ContainerDied","Data":"4bf65fdeceb059f8953c2bafd096a817979b64c3ea0c8edfa5b345d418acdd79"} Jun 15 06:48:44.634505 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.634132 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg" Jun 15 06:48:44.634505 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.634141 2569 scope.go:117] "RemoveContainer" containerID="a32561964cba0f3d0afebc149d764b236472d97bce39d0faddac1898efe87715" Jun 15 06:48:44.643008 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.642985 2569 scope.go:117] "RemoveContainer" containerID="f49e411f65fc8911b341e11db05c5f894c18fc22e2e1fe70fa23dbaaa0333b3f" Jun 15 06:48:44.651113 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.651093 2569 scope.go:117] "RemoveContainer" containerID="34d3223157f99e605587e958adce6e67e80bc34be5bb2385656af21d7bd778e4" Jun 15 06:48:44.656219 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.656153 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg"] Jun 15 06:48:44.658516 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:44.658492 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-67b657cbf5wbjmg"] Jun 15 06:48:46.056552 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:46.056508 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" path="/var/lib/kubelet/pods/fdf25b94-5830-4064-89ad-ac02b4a46325/volumes" Jun 15 06:48:49.569867 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:49.569752 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:48:49.592502 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:49.573103 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:48:55.016408 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016368 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7"] Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016786 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016800 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016817 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="llm-d-routing-sidecar" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016824 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="llm-d-routing-sidecar" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016832 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="storage-initializer" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016839 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="storage-initializer" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016846 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016851 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016864 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="storage-initializer" Jun 15 06:48:55.016868 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016869 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="storage-initializer" Jun 15 06:48:55.017207 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016924 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="llm-d-routing-sidecar" Jun 15 06:48:55.017207 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016934 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdf25b94-5830-4064-89ad-ac02b4a46325" containerName="main" Jun 15 06:48:55.017207 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.016940 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="4babe17d-2700-4b97-a9f4-d5fc4bb7045f" containerName="main" Jun 15 06:48:55.020155 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.020130 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.022643 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.022401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-4gxgm\"" Jun 15 06:48:55.023128 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.023093 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\"" Jun 15 06:48:55.026135 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.026109 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss"] Jun 15 06:48:55.032334 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.032308 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.032457 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.032405 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7"] Jun 15 06:48:55.046209 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.046178 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss"] Jun 15 06:48:55.070506 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.070467 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:48:55.099565 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099503 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.099745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.099745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099659 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.099745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099698 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e70f31f-219c-433b-be30-780a7806a56a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.099745 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099734 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.099890 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-home\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.099890 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099808 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.099890 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099848 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-dshm\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.099890 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.100015 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099929 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqwq\" (UniqueName: \"kubernetes.io/projected/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kube-api-access-5mqwq\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.100015 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099970 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.100015 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.099994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsf4s\" (UniqueName: \"kubernetes.io/projected/7e70f31f-219c-433b-be30-780a7806a56a-kube-api-access-fsf4s\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.100109 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.100081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.100146 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.100133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.200832 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.200789 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.200832 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.200834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-home\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201101 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.200861 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201101 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.200893 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-dshm\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201101 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.200924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201101 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.200960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqwq\" (UniqueName: \"kubernetes.io/projected/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kube-api-access-5mqwq\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201325 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201325 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsf4s\" (UniqueName: \"kubernetes.io/projected/7e70f31f-219c-433b-be30-780a7806a56a-kube-api-access-fsf4s\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201325 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201239 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201325 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201263 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201325 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201289 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201332 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201355 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-home\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201479 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e70f31f-219c-433b-be30-780a7806a56a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201534 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.201616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.202001 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201719 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.202001 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201759 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.202001 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.201810 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.203589 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.203527 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-dshm\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.203701 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.203608 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.203701 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.203678 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.204077 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.204056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e70f31f-219c-433b-be30-780a7806a56a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.209312 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.209290 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqwq\" (UniqueName: \"kubernetes.io/projected/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kube-api-access-5mqwq\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.209524 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.209503 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsf4s\" (UniqueName: \"kubernetes.io/projected/7e70f31f-219c-433b-be30-780a7806a56a-kube-api-access-fsf4s\") pod \"custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.333757 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.333716 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.345261 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.345225 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:48:55.434078 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.434008 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:48:55.502482 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.502448 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7"] Jun 15 06:48:55.505575 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:48:55.505525 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e70f31f_219c_433b_be30_780a7806a56a.slice/crio-facfd96c453e310e0a8d07e24ea80462d3cb68c628992a68c0d4cad52533f0ce WatchSource:0}: Error finding container facfd96c453e310e0a8d07e24ea80462d3cb68c628992a68c0d4cad52533f0ce: Status 404 returned error can't find the container with id facfd96c453e310e0a8d07e24ea80462d3cb68c628992a68c0d4cad52533f0ce Jun 15 06:48:55.522343 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.522316 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss"] Jun 15 06:48:55.524931 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:48:55.524900 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234c9116_9b8d_44e3_ab1a_4f8fbebaf541.slice/crio-9112d3e652c87c6d8264a75ca357f319342c516781b9b82c0105a21e9c130e73 WatchSource:0}: Error finding container 9112d3e652c87c6d8264a75ca357f319342c516781b9b82c0105a21e9c130e73: Status 404 returned error can't find the container with id 9112d3e652c87c6d8264a75ca357f319342c516781b9b82c0105a21e9c130e73 Jun 15 06:48:55.691282 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.691186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerStarted","Data":"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876"} Jun 15 06:48:55.691282 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.691251 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerStarted","Data":"facfd96c453e310e0a8d07e24ea80462d3cb68c628992a68c0d4cad52533f0ce"} Jun 15 06:48:55.691515 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.691345 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:48:55.694505 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.694469 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" event={"ID":"234c9116-9b8d-44e3-ab1a-4f8fbebaf541","Type":"ContainerStarted","Data":"a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f"} Jun 15 06:48:55.694753 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:55.694728 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" event={"ID":"234c9116-9b8d-44e3-ab1a-4f8fbebaf541","Type":"ContainerStarted","Data":"9112d3e652c87c6d8264a75ca357f319342c516781b9b82c0105a21e9c130e73"} Jun 15 06:48:56.702125 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:48:56.702084 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerStarted","Data":"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18"} Jun 15 06:49:00.726861 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:00.726833 2569 generic.go:356] "Generic (PLEG): container finished" podID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerID="a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f" exitCode=0 Jun 15 06:49:00.727261 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:00.726909 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" event={"ID":"234c9116-9b8d-44e3-ab1a-4f8fbebaf541","Type":"ContainerDied","Data":"a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f"} Jun 15 06:49:00.728819 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:00.728797 2569 generic.go:356] "Generic (PLEG): container finished" podID="7e70f31f-219c-433b-be30-780a7806a56a" containerID="375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18" exitCode=0 Jun 15 06:49:00.728910 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:00.728846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerDied","Data":"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18"} Jun 15 06:49:01.738684 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:01.738647 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerStarted","Data":"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce"} Jun 15 06:49:01.741615 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:01.741574 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" event={"ID":"234c9116-9b8d-44e3-ab1a-4f8fbebaf541","Type":"ContainerStarted","Data":"6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773"} Jun 15 06:49:01.763512 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:01.763444 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podStartSLOduration=2.822227457 podStartE2EDuration="7.763428102s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.941200645s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:49:01.762222277 +0000 UTC m=+3620.107425434" watchObservedRunningTime="2026-06-15 06:49:01.763428102 +0000 UTC m=+3620.108631244" Jun 15 06:49:01.784122 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:01.784056 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podStartSLOduration=1.9894314629999998 podStartE2EDuration="6.784034162s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.794602699s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:49:01.781955011 +0000 UTC m=+3620.127158151" watchObservedRunningTime="2026-06-15 06:49:01.784034162 +0000 UTC m=+3620.129237305" Jun 15 06:49:05.334101 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:05.334059 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:49:05.334731 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:05.334154 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:49:05.335459 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:05.335417 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:49:05.346044 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:05.346013 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:49:05.346287 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:05.346264 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:49:05.347278 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:05.347248 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:49:05.352394 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:05.352373 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:49:09.578407 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:09.578363 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:49:15.334490 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:15.334429 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:49:15.346318 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:15.346268 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:49:25.335047 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:25.334999 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:49:25.347005 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:25.346953 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:49:25.728684 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:25.728595 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:49:25.783653 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:25.783608 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:49:35.334907 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:35.334856 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:49:35.346489 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:35.346444 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:49:45.334729 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:45.334660 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:49:45.345613 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:45.345576 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:49:55.334199 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:55.334143 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:49:55.346005 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:49:55.345960 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:50:05.334271 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:05.334162 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:50:05.346512 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:05.346460 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:50:15.334529 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:15.334478 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:50:15.346228 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:15.346189 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:50:25.335133 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:25.335074 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:50:25.345602 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:25.345571 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:50:35.334881 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:35.334823 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:50:35.346373 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:35.346325 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:50:45.335048 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:45.334996 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:50:45.346077 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:45.346036 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:50:55.334410 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:55.334359 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:50:55.345616 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:50:55.345582 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:51:05.334251 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:05.334195 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:51:05.346094 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:05.346049 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:51:15.334576 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:15.334503 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:51:15.346763 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:15.346711 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:51:25.334309 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:25.334260 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:51:25.345674 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:25.345636 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:51:35.334867 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:35.334756 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8001/health\": dial tcp 10.134.0.46:8001: connect: connection refused" Jun 15 06:51:35.345720 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:35.345682 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused" Jun 15 06:51:45.345742 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:45.345706 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:51:45.355883 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:45.355854 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:51:45.363936 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:45.363907 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:51:45.364279 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:45.364262 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:51:50.970944 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:50.970908 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss"] Jun 15 06:51:50.974649 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:50.974615 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7"] Jun 15 06:51:50.994269 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:50.994233 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:51:51.051394 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:51:51.051360 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:52:05.999777 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:05.999730 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" containerID="cri-o://6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773" gracePeriod=45 Jun 15 06:52:06.002814 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.002768 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" containerID="cri-o://e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce" gracePeriod=45 Jun 15 06:52:06.819578 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.819528 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="llm-d-routing-sidecar" containerID="cri-o://3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876" gracePeriod=45 Jun 15 06:52:06.967717 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.967692 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:52:06.994123 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.993928 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mqwq\" (UniqueName: \"kubernetes.io/projected/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kube-api-access-5mqwq\") pod \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " Jun 15 06:52:06.994399 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.994333 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-home\") pod \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " Jun 15 06:52:06.994399 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.994387 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tls-certs\") pod \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " Jun 15 06:52:06.994593 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.994415 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tmp-dir\") pod \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " Jun 15 06:52:06.994593 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.994452 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-dshm\") pod \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " Jun 15 06:52:06.994593 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.994535 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kserve-provision-location\") pod \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " Jun 15 06:52:06.994771 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.994630 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-model-cache\") pod \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\" (UID: \"234c9116-9b8d-44e3-ab1a-4f8fbebaf541\") " Jun 15 06:52:06.995280 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.995036 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-home" (OuterVolumeSpecName: "home") pod "234c9116-9b8d-44e3-ab1a-4f8fbebaf541" (UID: "234c9116-9b8d-44e3-ab1a-4f8fbebaf541"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:06.997687 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.995711 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-model-cache" (OuterVolumeSpecName: "model-cache") pod "234c9116-9b8d-44e3-ab1a-4f8fbebaf541" (UID: "234c9116-9b8d-44e3-ab1a-4f8fbebaf541"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:06.998792 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.998747 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-dshm" (OuterVolumeSpecName: "dshm") pod "234c9116-9b8d-44e3-ab1a-4f8fbebaf541" (UID: "234c9116-9b8d-44e3-ab1a-4f8fbebaf541"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:06.999433 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:06.999405 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kube-api-access-5mqwq" (OuterVolumeSpecName: "kube-api-access-5mqwq") pod "234c9116-9b8d-44e3-ab1a-4f8fbebaf541" (UID: "234c9116-9b8d-44e3-ab1a-4f8fbebaf541"). InnerVolumeSpecName "kube-api-access-5mqwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:52:07.002235 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.002211 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "234c9116-9b8d-44e3-ab1a-4f8fbebaf541" (UID: "234c9116-9b8d-44e3-ab1a-4f8fbebaf541"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:52:07.015426 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.015384 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "234c9116-9b8d-44e3-ab1a-4f8fbebaf541" (UID: "234c9116-9b8d-44e3-ab1a-4f8fbebaf541"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:07.059474 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.059432 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "234c9116-9b8d-44e3-ab1a-4f8fbebaf541" (UID: "234c9116-9b8d-44e3-ab1a-4f8fbebaf541"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:07.086747 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.086718 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:52:07.095303 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095280 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-kserve-provision-location\") pod \"7e70f31f-219c-433b-be30-780a7806a56a\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " Jun 15 06:52:07.095457 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095322 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-tmp-dir\") pod \"7e70f31f-219c-433b-be30-780a7806a56a\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " Jun 15 06:52:07.095457 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095352 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-home\") pod \"7e70f31f-219c-433b-be30-780a7806a56a\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " Jun 15 06:52:07.095457 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095434 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsf4s\" (UniqueName: \"kubernetes.io/projected/7e70f31f-219c-433b-be30-780a7806a56a-kube-api-access-fsf4s\") pod \"7e70f31f-219c-433b-be30-780a7806a56a\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " Jun 15 06:52:07.095670 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095462 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-model-cache\") pod \"7e70f31f-219c-433b-be30-780a7806a56a\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " Jun 15 06:52:07.095670 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095493 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-dshm\") pod \"7e70f31f-219c-433b-be30-780a7806a56a\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " Jun 15 06:52:07.095670 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095605 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e70f31f-219c-433b-be30-780a7806a56a-tls-certs\") pod \"7e70f31f-219c-433b-be30-780a7806a56a\" (UID: \"7e70f31f-219c-433b-be30-780a7806a56a\") " Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095924 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mqwq\" (UniqueName: \"kubernetes.io/projected/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kube-api-access-5mqwq\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095945 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095960 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095976 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095988 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.095999 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.096010 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/234c9116-9b8d-44e3-ab1a-4f8fbebaf541-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.096097 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e70f31f-219c-433b-be30-780a7806a56a" (UID: "7e70f31f-219c-433b-be30-780a7806a56a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:07.096129 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.096101 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-home" (OuterVolumeSpecName: "home") pod "7e70f31f-219c-433b-be30-780a7806a56a" (UID: "7e70f31f-219c-433b-be30-780a7806a56a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:07.098162 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.098133 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e70f31f-219c-433b-be30-780a7806a56a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e70f31f-219c-433b-be30-780a7806a56a" (UID: "7e70f31f-219c-433b-be30-780a7806a56a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 06:52:07.098485 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.098461 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e70f31f-219c-433b-be30-780a7806a56a-kube-api-access-fsf4s" (OuterVolumeSpecName: "kube-api-access-fsf4s") pod "7e70f31f-219c-433b-be30-780a7806a56a" (UID: "7e70f31f-219c-433b-be30-780a7806a56a"). InnerVolumeSpecName "kube-api-access-fsf4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 06:52:07.098769 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.098746 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-dshm" (OuterVolumeSpecName: "dshm") pod "7e70f31f-219c-433b-be30-780a7806a56a" (UID: "7e70f31f-219c-433b-be30-780a7806a56a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:07.114665 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.114557 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e70f31f-219c-433b-be30-780a7806a56a" (UID: "7e70f31f-219c-433b-be30-780a7806a56a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:07.126265 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.126231 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e70f31f-219c-433b-be30-780a7806a56a" (UID: "7e70f31f-219c-433b-be30-780a7806a56a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 06:52:07.196777 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.196745 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e70f31f-219c-433b-be30-780a7806a56a-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.196777 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.196772 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.196777 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.196782 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.197012 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.196792 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.197012 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.196801 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsf4s\" (UniqueName: \"kubernetes.io/projected/7e70f31f-219c-433b-be30-780a7806a56a-kube-api-access-fsf4s\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.197012 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.196808 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.197012 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.196817 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e70f31f-219c-433b-be30-780a7806a56a-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 06:52:07.806498 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.806460 2569 generic.go:356] "Generic (PLEG): container finished" podID="7e70f31f-219c-433b-be30-780a7806a56a" containerID="e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce" exitCode=0 Jun 15 06:52:07.806498 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.806494 2569 generic.go:356] "Generic (PLEG): container finished" podID="7e70f31f-219c-433b-be30-780a7806a56a" containerID="3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876" exitCode=0 Jun 15 06:52:07.806756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.806495 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerDied","Data":"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce"} Jun 15 06:52:07.806756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.806516 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" Jun 15 06:52:07.806756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.806586 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerDied","Data":"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876"} Jun 15 06:52:07.806756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.806614 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7" event={"ID":"7e70f31f-219c-433b-be30-780a7806a56a","Type":"ContainerDied","Data":"facfd96c453e310e0a8d07e24ea80462d3cb68c628992a68c0d4cad52533f0ce"} Jun 15 06:52:07.806756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.806643 2569 scope.go:117] "RemoveContainer" containerID="e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce" Jun 15 06:52:07.808583 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.808528 2569 generic.go:356] "Generic (PLEG): container finished" podID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerID="6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773" exitCode=0 Jun 15 06:52:07.808583 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.808573 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" event={"ID":"234c9116-9b8d-44e3-ab1a-4f8fbebaf541","Type":"ContainerDied","Data":"6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773"} Jun 15 06:52:07.808793 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.808628 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" event={"ID":"234c9116-9b8d-44e3-ab1a-4f8fbebaf541","Type":"ContainerDied","Data":"9112d3e652c87c6d8264a75ca357f319342c516781b9b82c0105a21e9c130e73"} Jun 15 06:52:07.808793 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.808638 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss" Jun 15 06:52:07.817704 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.817685 2569 scope.go:117] "RemoveContainer" containerID="375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18" Jun 15 06:52:07.827902 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.827876 2569 scope.go:117] "RemoveContainer" containerID="3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876" Jun 15 06:52:07.832628 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.832601 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7"] Jun 15 06:52:07.837865 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.837838 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-78b45dc7ff-nzkk7"] Jun 15 06:52:07.838954 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.838932 2569 scope.go:117] "RemoveContainer" containerID="e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce" Jun 15 06:52:07.839240 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:52:07.839222 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce\": container with ID starting with e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce not found: ID does not exist" containerID="e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce" Jun 15 06:52:07.839315 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.839248 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce"} err="failed to get container status \"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce\": rpc error: code = NotFound desc = could not find container \"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce\": container with ID starting with e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce not found: ID does not exist" Jun 15 06:52:07.839315 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.839264 2569 scope.go:117] "RemoveContainer" containerID="375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18" Jun 15 06:52:07.839515 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:52:07.839499 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18\": container with ID starting with 375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18 not found: ID does not exist" containerID="375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18" Jun 15 06:52:07.839603 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.839518 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18"} err="failed to get container status \"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18\": rpc error: code = NotFound desc = could not find container \"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18\": container with ID starting with 375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18 not found: ID does not exist" Jun 15 06:52:07.839603 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.839530 2569 scope.go:117] "RemoveContainer" containerID="3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876" Jun 15 06:52:07.839814 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:52:07.839797 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876\": container with ID starting with 3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876 not found: ID does not exist" containerID="3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876" Jun 15 06:52:07.839859 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.839818 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876"} err="failed to get container status \"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876\": rpc error: code = NotFound desc = could not find container \"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876\": container with ID starting with 3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876 not found: ID does not exist" Jun 15 06:52:07.839859 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.839833 2569 scope.go:117] "RemoveContainer" containerID="e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce" Jun 15 06:52:07.840092 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.840072 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce"} err="failed to get container status \"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce\": rpc error: code = NotFound desc = could not find container \"e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce\": container with ID starting with e415076a176726ccd404fafcfb50442ce4f2032aa700e3d85051e33191aab1ce not found: ID does not exist" Jun 15 06:52:07.840143 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.840095 2569 scope.go:117] "RemoveContainer" containerID="375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18" Jun 15 06:52:07.840312 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.840291 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18"} err="failed to get container status \"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18\": rpc error: code = NotFound desc = could not find container \"375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18\": container with ID starting with 375c4195a054b6b91ab6b83130cacca424575244cd970c6cf6d6f1602cd4df18 not found: ID does not exist" Jun 15 06:52:07.840368 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.840314 2569 scope.go:117] "RemoveContainer" containerID="3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876" Jun 15 06:52:07.840561 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.840512 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876"} err="failed to get container status \"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876\": rpc error: code = NotFound desc = could not find container \"3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876\": container with ID starting with 3cd409573739443da5a1681045c485bbf4eaa04d85781fd5b8ff83eac042a876 not found: ID does not exist" Jun 15 06:52:07.840561 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.840554 2569 scope.go:117] "RemoveContainer" containerID="6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773" Jun 15 06:52:07.848809 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.848786 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss"] Jun 15 06:52:07.848968 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.848954 2569 scope.go:117] "RemoveContainer" containerID="a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f" Jun 15 06:52:07.852811 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.852787 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-7b4cdcb48f-wnvss"] Jun 15 06:52:07.857222 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.857205 2569 scope.go:117] "RemoveContainer" containerID="6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773" Jun 15 06:52:07.857478 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:52:07.857452 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773\": container with ID starting with 6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773 not found: ID does not exist" containerID="6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773" Jun 15 06:52:07.857525 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.857484 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773"} err="failed to get container status \"6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773\": rpc error: code = NotFound desc = could not find container \"6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773\": container with ID starting with 6c7d8cdfe482022439faa7b94a92e41f1097c175fc8fa33df908a64c2aad7773 not found: ID does not exist" Jun 15 06:52:07.857525 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.857499 2569 scope.go:117] "RemoveContainer" containerID="a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f" Jun 15 06:52:07.857761 ip-10-0-128-226 kubenswrapper[2569]: E0615 06:52:07.857741 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f\": container with ID starting with a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f not found: ID does not exist" containerID="a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f" Jun 15 06:52:07.857844 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:07.857768 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f"} err="failed to get container status \"a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f\": rpc error: code = NotFound desc = could not find container \"a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f\": container with ID starting with a713702b6dda84017adf8d54a83e4b09213f5cdf186b22145bb599486c1f5b8f not found: ID does not exist" Jun 15 06:52:08.057959 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:08.057923 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" path="/var/lib/kubelet/pods/234c9116-9b8d-44e3-ab1a-4f8fbebaf541/volumes" Jun 15 06:52:08.058384 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:08.058370 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e70f31f-219c-433b-be30-780a7806a56a" path="/var/lib/kubelet/pods/7e70f31f-219c-433b-be30-780a7806a56a/volumes" Jun 15 06:52:11.696517 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:11.696481 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:52:11.794662 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:11.794626 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:52:15.697274 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:15.697238 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:52:20.803362 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:20.803316 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:52:26.238684 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.238645 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4"] Jun 15 06:52:26.239220 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239198 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="storage-initializer" Jun 15 06:52:26.239304 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239223 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="storage-initializer" Jun 15 06:52:26.239304 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239237 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="storage-initializer" Jun 15 06:52:26.239304 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239248 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="storage-initializer" Jun 15 06:52:26.239304 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239270 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" Jun 15 06:52:26.239304 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239279 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" Jun 15 06:52:26.239304 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239303 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="llm-d-routing-sidecar" Jun 15 06:52:26.239635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239312 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="llm-d-routing-sidecar" Jun 15 06:52:26.239635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239334 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" Jun 15 06:52:26.239635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239342 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" Jun 15 06:52:26.239635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239431 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="234c9116-9b8d-44e3-ab1a-4f8fbebaf541" containerName="main" Jun 15 06:52:26.239635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239446 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="llm-d-routing-sidecar" Jun 15 06:52:26.239635 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.239457 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e70f31f-219c-433b-be30-780a7806a56a" containerName="main" Jun 15 06:52:26.242800 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.242779 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.245200 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.245176 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-nt6r8\"" Jun 15 06:52:26.245295 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.245221 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\"" Jun 15 06:52:26.253979 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.253949 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4"] Jun 15 06:52:26.336253 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.336215 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp"] Jun 15 06:52:26.340498 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.340475 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.351144 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.351119 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp"] Jun 15 06:52:26.364344 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.364318 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:52:26.375894 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.375868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.376136 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.376113 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-dshm\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.376241 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.376222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-home\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.376314 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.376291 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-model-cache\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.376371 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.376335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2zn9\" (UniqueName: \"kubernetes.io/projected/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-kube-api-access-s2zn9\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.376425 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.376374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.376478 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.376461 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-tls-certs\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477214 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-tls-certs\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477418 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477225 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.477418 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c26092a-3ad8-4fc2-879a-4b96a47c9166-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.477535 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477444 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcvt\" (UniqueName: \"kubernetes.io/projected/9c26092a-3ad8-4fc2-879a-4b96a47c9166-kube-api-access-kfcvt\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.477535 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477503 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477535 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477530 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-dshm\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477713 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-home\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.477713 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-home\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477821 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.477877 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477821 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-model-cache\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477877 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.477994 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-tmp-dir\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477994 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477949 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2zn9\" (UniqueName: \"kubernetes.io/projected/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-kube-api-access-s2zn9\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.477994 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.477981 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-home\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.478139 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.478007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.478139 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.478039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.478229 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.478158 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-model-cache\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.478293 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.478276 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.480112 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.480089 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-dshm\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.480370 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.480351 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-tls-certs\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.487123 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.487098 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2zn9\" (UniqueName: \"kubernetes.io/projected/5d2cc4b0-4097-4d57-9116-4a62c41cc76d-kube-api-access-s2zn9\") pod \"router-with-refs-pd-test-kserve-6f78896447-wshh4\" (UID: \"5d2cc4b0-4097-4d57-9116-4a62c41cc76d\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.554014 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.553978 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:26.579505 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.579472 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.579692 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.579516 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.579692 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.579569 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.579692 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.579627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.579692 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.579652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c26092a-3ad8-4fc2-879a-4b96a47c9166-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.579910 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.579693 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcvt\" (UniqueName: \"kubernetes.io/projected/9c26092a-3ad8-4fc2-879a-4b96a47c9166-kube-api-access-kfcvt\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.579910 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.579846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-home\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.580035 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.580011 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.580094 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.580036 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.580094 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.580071 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.580409 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.580388 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-home\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.581852 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.581830 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c26092a-3ad8-4fc2-879a-4b96a47c9166-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.582667 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.582646 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c26092a-3ad8-4fc2-879a-4b96a47c9166-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.589510 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.589480 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcvt\" (UniqueName: \"kubernetes.io/projected/9c26092a-3ad8-4fc2-879a-4b96a47c9166-kube-api-access-kfcvt\") pod \"router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp\" (UID: \"9c26092a-3ad8-4fc2-879a-4b96a47c9166\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.651961 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.651907 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:52:26.700245 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.700139 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4"] Jun 15 06:52:26.702632 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:52:26.702605 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2cc4b0_4097_4d57_9116_4a62c41cc76d.slice/crio-2280a46663c9c872ce2706b35e333d2197bbe4e76bc92f6e79077ac2293757e7 WatchSource:0}: Error finding container 2280a46663c9c872ce2706b35e333d2197bbe4e76bc92f6e79077ac2293757e7: Status 404 returned error can't find the container with id 2280a46663c9c872ce2706b35e333d2197bbe4e76bc92f6e79077ac2293757e7 Jun 15 06:52:26.801314 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.801281 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp"] Jun 15 06:52:26.803310 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:52:26.803283 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c26092a_3ad8_4fc2_879a_4b96a47c9166.slice/crio-9755a630b94e3330f8f738302f2239e2801684e39d60dfedb5e16b5b44d466cd WatchSource:0}: Error finding container 9755a630b94e3330f8f738302f2239e2801684e39d60dfedb5e16b5b44d466cd: Status 404 returned error can't find the container with id 9755a630b94e3330f8f738302f2239e2801684e39d60dfedb5e16b5b44d466cd Jun 15 06:52:26.909097 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.909049 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" event={"ID":"9c26092a-3ad8-4fc2-879a-4b96a47c9166","Type":"ContainerStarted","Data":"b8415f513b0225674e4ba57b4ebab3ffddda0b02f5a7f22ff52ec88b94a4e1ef"} Jun 15 06:52:26.909297 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.909125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" event={"ID":"9c26092a-3ad8-4fc2-879a-4b96a47c9166","Type":"ContainerStarted","Data":"9755a630b94e3330f8f738302f2239e2801684e39d60dfedb5e16b5b44d466cd"} Jun 15 06:52:26.910795 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.910771 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" event={"ID":"5d2cc4b0-4097-4d57-9116-4a62c41cc76d","Type":"ContainerStarted","Data":"2c5583e9d57e610617e4afe818b762c966b02171e8da1bb9cb31b6b6c39db1bc"} Jun 15 06:52:26.910902 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.910813 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" event={"ID":"5d2cc4b0-4097-4d57-9116-4a62c41cc76d","Type":"ContainerStarted","Data":"2280a46663c9c872ce2706b35e333d2197bbe4e76bc92f6e79077ac2293757e7"} Jun 15 06:52:26.910902 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:26.910834 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:27.917879 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:27.917839 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" event={"ID":"5d2cc4b0-4097-4d57-9116-4a62c41cc76d","Type":"ContainerStarted","Data":"05b64457d70b2f1f5aced8df47a0ac28d696157ad85c665ae991a14efd8d41c3"} Jun 15 06:52:28.339199 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:28.339152 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:52:31.952785 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:31.952694 2569 generic.go:356] "Generic (PLEG): container finished" podID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerID="05b64457d70b2f1f5aced8df47a0ac28d696157ad85c665ae991a14efd8d41c3" exitCode=0 Jun 15 06:52:31.952785 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:31.952743 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" event={"ID":"5d2cc4b0-4097-4d57-9116-4a62c41cc76d","Type":"ContainerDied","Data":"05b64457d70b2f1f5aced8df47a0ac28d696157ad85c665ae991a14efd8d41c3"} Jun 15 06:52:32.960735 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:32.960703 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" event={"ID":"5d2cc4b0-4097-4d57-9116-4a62c41cc76d","Type":"ContainerStarted","Data":"d68822c89ab4c22d8c6af65fd68d08df9f1c852252b8d75890f4192af6f26a69"} Jun 15 06:52:32.986305 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:32.986235 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podStartSLOduration=2.394524585 podStartE2EDuration="6.986214993s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.591690408s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:52:32.985808514 +0000 UTC m=+3831.331011693" watchObservedRunningTime="2026-06-15 06:52:32.986214993 +0000 UTC m=+3831.331418125" Jun 15 06:52:36.554802 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:36.554765 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:36.555256 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:36.554840 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:36.556167 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:36.556133 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:52:36.574044 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:36.574005 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:52:46.555260 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:46.555119 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:52:56.554601 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:52:56.554553 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:53:06.555333 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:06.555224 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:53:16.555435 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:16.555384 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:53:18.227284 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:18.227248 2569 generic.go:356] "Generic (PLEG): container finished" podID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerID="b8415f513b0225674e4ba57b4ebab3ffddda0b02f5a7f22ff52ec88b94a4e1ef" exitCode=0 Jun 15 06:53:18.227700 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:18.227287 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" event={"ID":"9c26092a-3ad8-4fc2-879a-4b96a47c9166","Type":"ContainerDied","Data":"b8415f513b0225674e4ba57b4ebab3ffddda0b02f5a7f22ff52ec88b94a4e1ef"} Jun 15 06:53:19.237857 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:19.237819 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" event={"ID":"9c26092a-3ad8-4fc2-879a-4b96a47c9166","Type":"ContainerStarted","Data":"fe63a26a8ca996b9f2ad0eb8a01417e5eee30c0abbe5167f727442ff8d283ce7"} Jun 15 06:53:19.261758 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:19.261686 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podStartSLOduration=2.031756471 podStartE2EDuration="53.261664781s" totalImagesPullingTime="0s" totalInitContainerRuntime="51.22990831s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:53:19.259905047 +0000 UTC m=+3877.605108189" watchObservedRunningTime="2026-06-15 06:53:19.261664781 +0000 UTC m=+3877.606867923" Jun 15 06:53:26.554552 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:26.554481 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:53:26.652323 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:26.652277 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:53:26.652499 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:26.652406 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:53:26.654059 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:26.654031 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:53:36.555117 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:36.555058 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:53:36.653086 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:36.653028 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:53:46.554923 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:46.554869 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:53:46.652516 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:46.652452 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:53:49.606409 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:49.606295 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:53:49.612087 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:49.612058 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:53:56.554733 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:56.554671 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:53:56.652756 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:53:56.652695 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:54:06.554460 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:06.554411 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:54:06.652612 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:06.652571 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:54:16.554675 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:16.554614 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:54:16.652721 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:16.652674 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:54:26.555518 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:26.555428 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" podUID="5d2cc4b0-4097-4d57-9116-4a62c41cc76d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8001/health\": dial tcp 10.134.0.48:8001: connect: connection refused" Jun 15 06:54:26.653293 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:26.653250 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:54:36.564203 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:36.564125 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:54:36.575865 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:36.575840 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-6f78896447-wshh4" Jun 15 06:54:36.652568 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:36.652498 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:54:46.652799 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:46.652753 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:54:55.777204 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:55.777104 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:54:56.652436 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:56.652388 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:54:56.968078 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:54:56.967989 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:55:03.285029 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:55:03.284987 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:55:06.652863 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:55:06.652812 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" podUID="9c26092a-3ad8-4fc2-879a-4b96a47c9166" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused" Jun 15 06:55:09.901735 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:55:09.901704 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:55:10.002878 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:55:10.002845 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:55:16.662446 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:55:16.662412 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:55:16.671273 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:55:16.671246 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp" Jun 15 06:58:49.640858 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:58:49.640746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:58:49.648310 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:58:49.648284 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 06:59:41.044313 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:41.044238 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 06:59:59.483243 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.483207 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8"] Jun 15 06:59:59.486998 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.486976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.491033 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.491011 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\"" Jun 15 06:59:59.516478 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.516437 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8"] Jun 15 06:59:59.562942 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.562895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.563142 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.562958 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.563142 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.562992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mk6p\" (UniqueName: \"kubernetes.io/projected/8572c5e6-3714-4917-b782-31ec52c11134-kube-api-access-9mk6p\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.563142 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.563063 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8572c5e6-3714-4917-b782-31ec52c11134-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.563321 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.563150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8572c5e6-3714-4917-b782-31ec52c11134-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.563321 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.563195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.563321 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.563278 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.563456 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.563337 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.569191 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.569155 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 06:59:59.664651 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664611 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.664912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664682 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.664912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.664912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664765 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.664912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mk6p\" (UniqueName: \"kubernetes.io/projected/8572c5e6-3714-4917-b782-31ec52c11134-kube-api-access-9mk6p\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.664912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664825 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8572c5e6-3714-4917-b782-31ec52c11134-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.664912 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664890 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8572c5e6-3714-4917-b782-31ec52c11134-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.665298 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.664926 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.665298 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.665205 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.665458 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.665432 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.665688 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.665467 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.665806 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.665570 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.666018 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.665994 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8572c5e6-3714-4917-b782-31ec52c11134-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.667283 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.667256 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8572c5e6-3714-4917-b782-31ec52c11134-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.667814 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.667790 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8572c5e6-3714-4917-b782-31ec52c11134-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.675022 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.674998 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mk6p\" (UniqueName: \"kubernetes.io/projected/8572c5e6-3714-4917-b782-31ec52c11134-kube-api-access-9mk6p\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8\" (UID: \"8572c5e6-3714-4917-b782-31ec52c11134\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.797339 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.797300 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 06:59:59.957074 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.957020 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8"] Jun 15 06:59:59.958409 ip-10-0-128-226 kubenswrapper[2569]: W0615 06:59:59.958380 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8572c5e6_3714_4917_b782_31ec52c11134.slice/crio-fa847b9be8473af928d2fa278a1c312a0bfb481ad5aca61d218824a9893abf51 WatchSource:0}: Error finding container fa847b9be8473af928d2fa278a1c312a0bfb481ad5aca61d218824a9893abf51: Status 404 returned error can't find the container with id fa847b9be8473af928d2fa278a1c312a0bfb481ad5aca61d218824a9893abf51 Jun 15 06:59:59.968708 ip-10-0-128-226 kubenswrapper[2569]: I0615 06:59:59.968684 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jun 15 07:00:00.418693 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:00.418653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" event={"ID":"8572c5e6-3714-4917-b782-31ec52c11134","Type":"ContainerStarted","Data":"e4b50a533016f1dcb35821769f2f190ad25dc25c902839f4a374e6efec3cd120"} Jun 15 07:00:00.418864 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:00.418722 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" event={"ID":"8572c5e6-3714-4917-b782-31ec52c11134","Type":"ContainerStarted","Data":"fa847b9be8473af928d2fa278a1c312a0bfb481ad5aca61d218824a9893abf51"} Jun 15 07:00:00.590011 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:00.589978 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:00:20.554060 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:20.554021 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:00:24.543409 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:24.543361 2569 generic.go:356] "Generic (PLEG): container finished" podID="8572c5e6-3714-4917-b782-31ec52c11134" containerID="e4b50a533016f1dcb35821769f2f190ad25dc25c902839f4a374e6efec3cd120" exitCode=0 Jun 15 07:00:24.543921 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:24.543440 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" event={"ID":"8572c5e6-3714-4917-b782-31ec52c11134","Type":"ContainerDied","Data":"e4b50a533016f1dcb35821769f2f190ad25dc25c902839f4a374e6efec3cd120"} Jun 15 07:00:25.551110 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:25.551075 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" event={"ID":"8572c5e6-3714-4917-b782-31ec52c11134","Type":"ContainerStarted","Data":"7e2a1e01bf22fe3caafc6483f1651317c4ec17ef08f1dcde0005e59880e149d0"} Jun 15 07:00:25.578337 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:25.578282 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podStartSLOduration=2.1923213 podStartE2EDuration="26.578266006s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.385944706s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:00:25.574069827 +0000 UTC m=+4303.919272987" watchObservedRunningTime="2026-06-15 07:00:25.578266006 +0000 UTC m=+4303.923469149" Jun 15 07:00:29.797697 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:29.797663 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 07:00:29.798208 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:29.797730 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 07:00:29.799702 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:29.799661 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:00:39.798678 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:39.798573 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:00:49.798035 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:49.797990 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:00:59.798155 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:00:59.798099 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:01:09.797760 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:09.797716 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:01:19.798620 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:19.798533 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:01:29.798048 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:29.797993 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:01:39.798479 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:39.798423 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" podUID="8572c5e6-3714-4917-b782-31ec52c11134" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused" Jun 15 07:01:49.808577 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:49.808517 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 07:01:49.820567 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:49.820524 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8" Jun 15 07:01:55.929386 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:55.929348 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:01:55.990386 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:01:55.990356 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:03:15.025666 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:03:15.025621 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:03:49.677586 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:03:49.677451 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:03:49.686685 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:03:49.686657 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:04:28.618187 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:04:28.618150 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:07:15.946476 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:15.946440 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:07:15.985335 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:15.985300 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:07:15.995818 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:15.995780 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:07:16.004369 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:16.004336 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:07:16.014724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:16.014668 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:07:34.058208 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.058179 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8"] Jun 15 07:07:34.061966 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.061943 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.064698 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.064674 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\"" Jun 15 07:07:34.080049 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.080023 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8"] Jun 15 07:07:34.085703 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.085679 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:07:34.233201 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.233157 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.233382 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.233247 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.233382 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.233280 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.233382 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.233308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.233526 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.233410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0058837-85ca-4164-b253-2290ecfcc950-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.233526 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.233460 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.233659 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.233528 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78gxf\" (UniqueName: \"kubernetes.io/projected/e0058837-85ca-4164-b253-2290ecfcc950-kube-api-access-78gxf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.334530 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.334431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.334530 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.334494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.334530 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.334515 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.334842 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.334555 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.334842 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.334638 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0058837-85ca-4164-b253-2290ecfcc950-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.334842 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.334686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.334842 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.334734 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78gxf\" (UniqueName: \"kubernetes.io/projected/e0058837-85ca-4164-b253-2290ecfcc950-kube-api-access-78gxf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.335055 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.335018 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.335114 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.335062 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.335398 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.335363 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.335531 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.335423 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.337256 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.337236 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.337486 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.337461 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0058837-85ca-4164-b253-2290ecfcc950-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.344216 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.344193 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78gxf\" (UniqueName: \"kubernetes.io/projected/e0058837-85ca-4164-b253-2290ecfcc950-kube-api-access-78gxf\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.374709 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.374678 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:34.529893 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.529854 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8"] Jun 15 07:07:34.531586 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:07:34.531556 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0058837_85ca_4164_b253_2290ecfcc950.slice/crio-2ac13c8045312c733e1a5bdba8d18a6a011917443f066cf5936da1babe03e49c WatchSource:0}: Error finding container 2ac13c8045312c733e1a5bdba8d18a6a011917443f066cf5936da1babe03e49c: Status 404 returned error can't find the container with id 2ac13c8045312c733e1a5bdba8d18a6a011917443f066cf5936da1babe03e49c Jun 15 07:07:34.536338 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.536310 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jun 15 07:07:34.900182 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.900080 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" event={"ID":"e0058837-85ca-4164-b253-2290ecfcc950","Type":"ContainerStarted","Data":"bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68"} Jun 15 07:07:34.900182 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:34.900171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" event={"ID":"e0058837-85ca-4164-b253-2290ecfcc950","Type":"ContainerStarted","Data":"2ac13c8045312c733e1a5bdba8d18a6a011917443f066cf5936da1babe03e49c"} Jun 15 07:07:51.000357 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:51.000319 2569 generic.go:356] "Generic (PLEG): container finished" podID="e0058837-85ca-4164-b253-2290ecfcc950" containerID="bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68" exitCode=0 Jun 15 07:07:51.000856 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:51.000392 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" event={"ID":"e0058837-85ca-4164-b253-2290ecfcc950","Type":"ContainerDied","Data":"bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68"} Jun 15 07:07:52.007318 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:52.007274 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" event={"ID":"e0058837-85ca-4164-b253-2290ecfcc950","Type":"ContainerStarted","Data":"52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81"} Jun 15 07:07:52.033017 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:52.032955 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podStartSLOduration=2.3587502750000002 podStartE2EDuration="18.032938557s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.674188282s" isStatefulPod=false podCreationTimestamp="2026-06-15 07:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:07:52.029226941 +0000 UTC m=+4750.374430102" watchObservedRunningTime="2026-06-15 07:07:52.032938557 +0000 UTC m=+4750.378141698" Jun 15 07:07:54.375101 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:54.375065 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:54.375477 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:54.375145 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:07:54.376624 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:07:54.376589 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:08:04.375465 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:04.375372 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:08:14.375236 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:14.375181 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:08:24.376143 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:24.376097 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:08:34.375972 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:34.375928 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:08:44.376042 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:44.375994 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:08:49.718607 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:49.718477 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:08:49.730693 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:49.730666 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:08:54.375509 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:08:54.375469 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:09:04.375817 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:04.375776 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:09:14.375648 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:14.375600 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:09:24.375260 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:24.375220 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:09:34.375157 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:34.375061 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:09:42.690415 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:42.690382 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:09:44.375972 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:44.375931 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused" Jun 15 07:09:54.385435 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:54.385401 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:09:54.393207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:54.393179 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:09:58.659013 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:58.658976 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8"] Jun 15 07:09:58.675657 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:58.675622 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:09:59.321944 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:09:59.321909 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:10:13.684697 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:13.684648 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" containerID="cri-o://52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81" gracePeriod=45 Jun 15 07:10:14.456067 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.456042 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:10:14.614615 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614569 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0058837-85ca-4164-b253-2290ecfcc950-tls-certs\") pod \"e0058837-85ca-4164-b253-2290ecfcc950\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " Jun 15 07:10:14.614824 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614669 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-home\") pod \"e0058837-85ca-4164-b253-2290ecfcc950\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " Jun 15 07:10:14.614824 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614700 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-dshm\") pod \"e0058837-85ca-4164-b253-2290ecfcc950\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " Jun 15 07:10:14.614824 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614738 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-model-cache\") pod \"e0058837-85ca-4164-b253-2290ecfcc950\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " Jun 15 07:10:14.614824 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614824 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-kserve-provision-location\") pod \"e0058837-85ca-4164-b253-2290ecfcc950\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " Jun 15 07:10:14.615117 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614884 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78gxf\" (UniqueName: \"kubernetes.io/projected/e0058837-85ca-4164-b253-2290ecfcc950-kube-api-access-78gxf\") pod \"e0058837-85ca-4164-b253-2290ecfcc950\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " Jun 15 07:10:14.615117 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614920 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-tmp-dir\") pod \"e0058837-85ca-4164-b253-2290ecfcc950\" (UID: \"e0058837-85ca-4164-b253-2290ecfcc950\") " Jun 15 07:10:14.615117 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.614987 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0058837-85ca-4164-b253-2290ecfcc950" (UID: "e0058837-85ca-4164-b253-2290ecfcc950"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:10:14.615286 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.615276 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:10:14.615575 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.615489 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-home" (OuterVolumeSpecName: "home") pod "e0058837-85ca-4164-b253-2290ecfcc950" (UID: "e0058837-85ca-4164-b253-2290ecfcc950"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:10:14.617359 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.617323 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-dshm" (OuterVolumeSpecName: "dshm") pod "e0058837-85ca-4164-b253-2290ecfcc950" (UID: "e0058837-85ca-4164-b253-2290ecfcc950"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:10:14.617471 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.617421 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0058837-85ca-4164-b253-2290ecfcc950-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0058837-85ca-4164-b253-2290ecfcc950" (UID: "e0058837-85ca-4164-b253-2290ecfcc950"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 07:10:14.617954 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.617928 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0058837-85ca-4164-b253-2290ecfcc950-kube-api-access-78gxf" (OuterVolumeSpecName: "kube-api-access-78gxf") pod "e0058837-85ca-4164-b253-2290ecfcc950" (UID: "e0058837-85ca-4164-b253-2290ecfcc950"). InnerVolumeSpecName "kube-api-access-78gxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 07:10:14.632321 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.632293 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0058837-85ca-4164-b253-2290ecfcc950" (UID: "e0058837-85ca-4164-b253-2290ecfcc950"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:10:14.641843 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.641805 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0058837-85ca-4164-b253-2290ecfcc950" (UID: "e0058837-85ca-4164-b253-2290ecfcc950"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:10:14.715845 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.715803 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:10:14.715845 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.715832 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78gxf\" (UniqueName: \"kubernetes.io/projected/e0058837-85ca-4164-b253-2290ecfcc950-kube-api-access-78gxf\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:10:14.715845 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.715842 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:10:14.715845 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.715852 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0058837-85ca-4164-b253-2290ecfcc950-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:10:14.716669 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.715864 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:10:14.716669 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.715871 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0058837-85ca-4164-b253-2290ecfcc950-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:10:14.836767 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.836731 2569 generic.go:356] "Generic (PLEG): container finished" podID="e0058837-85ca-4164-b253-2290ecfcc950" containerID="52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81" exitCode=0 Jun 15 07:10:14.836941 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.836795 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" Jun 15 07:10:14.836941 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.836815 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" event={"ID":"e0058837-85ca-4164-b253-2290ecfcc950","Type":"ContainerDied","Data":"52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81"} Jun 15 07:10:14.836941 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.836878 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" event={"ID":"e0058837-85ca-4164-b253-2290ecfcc950","Type":"ContainerDied","Data":"2ac13c8045312c733e1a5bdba8d18a6a011917443f066cf5936da1babe03e49c"} Jun 15 07:10:14.836941 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.836908 2569 scope.go:117] "RemoveContainer" containerID="52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81" Jun 15 07:10:14.855947 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.855902 2569 scope.go:117] "RemoveContainer" containerID="bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68" Jun 15 07:10:14.865477 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.865452 2569 scope.go:117] "RemoveContainer" containerID="52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81" Jun 15 07:10:14.866202 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:10:14.866043 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81\": container with ID starting with 52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81 not found: ID does not exist" containerID="52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81" Jun 15 07:10:14.866202 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.866187 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81"} err="failed to get container status \"52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81\": rpc error: code = NotFound desc = could not find container \"52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81\": container with ID starting with 52c6ab569a571b0aae78e6656f765321ffd43b02266e8c0f90d14e1152931d81 not found: ID does not exist" Jun 15 07:10:14.866475 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.866225 2569 scope.go:117] "RemoveContainer" containerID="bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68" Jun 15 07:10:14.866715 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:10:14.866680 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68\": container with ID starting with bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68 not found: ID does not exist" containerID="bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68" Jun 15 07:10:14.867262 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.866722 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68"} err="failed to get container status \"bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68\": rpc error: code = NotFound desc = could not find container \"bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68\": container with ID starting with bf9d7085a626dbd7460e22cca862558eb480424efdc72e0aecd4d952f05abc68 not found: ID does not exist" Jun 15 07:10:14.869371 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.869347 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8"] Jun 15 07:10:14.874506 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:14.874485 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8"] Jun 15 07:10:16.057823 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:16.057789 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0058837-85ca-4164-b253-2290ecfcc950" path="/var/lib/kubelet/pods/e0058837-85ca-4164-b253-2290ecfcc950/volumes" Jun 15 07:10:19.386395 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:19.386354 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-55868c9d74fpbf8" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jun 15 07:10:29.651816 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:29.651752 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:10:29.763728 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:29.763694 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:10:29.935686 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:29.935597 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:10:30.007487 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.007453 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:10:30.009158 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.009129 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x"] Jun 15 07:10:30.009784 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.009765 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" Jun 15 07:10:30.009844 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.009788 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" Jun 15 07:10:30.009844 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.009803 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="storage-initializer" Jun 15 07:10:30.009844 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.009814 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="storage-initializer" Jun 15 07:10:30.009947 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.009918 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0058837-85ca-4164-b253-2290ecfcc950" containerName="main" Jun 15 07:10:30.015724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.015705 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.021185 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.021162 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-dbx9g\"" Jun 15 07:10:30.021470 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.021451 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\"" Jun 15 07:10:30.044978 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.044944 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Jun 15 07:10:30.048706 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.048686 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.051969 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.051949 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-sxhgn\"" Jun 15 07:10:30.060589 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.060528 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x"] Jun 15 07:10:30.060934 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.060914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.061013 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.060995 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.061077 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.061043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.061181 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.061147 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-cache\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.061254 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.061229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-tmp\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.061313 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.061281 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmtr2\" (UniqueName: \"kubernetes.io/projected/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kube-api-access-mmtr2\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.077568 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.077507 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Jun 15 07:10:30.162656 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162613 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.162856 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.162856 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.162856 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.162856 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08841648-1c7b-4c49-8701-dc9289afa4ad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.162856 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162823 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-cache\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.162856 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162850 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.163207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162889 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.163207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-tmp\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.163207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.162951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmtr2\" (UniqueName: \"kubernetes.io/projected/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kube-api-access-mmtr2\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.163207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.163106 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.163207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.163130 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.163207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.163180 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-cache\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.163207 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.163184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.163619 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.163267 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-tmp\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.163619 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.163327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.163619 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.163374 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnrhr\" (UniqueName: \"kubernetes.io/projected/08841648-1c7b-4c49-8701-dc9289afa4ad-kube-api-access-dnrhr\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.166101 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.166074 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.172158 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.172130 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmtr2\" (UniqueName: \"kubernetes.io/projected/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kube-api-access-mmtr2\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.264123 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264017 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnrhr\" (UniqueName: \"kubernetes.io/projected/08841648-1c7b-4c49-8701-dc9289afa4ad-kube-api-access-dnrhr\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264123 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264064 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264123 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264112 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264420 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264145 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08841648-1c7b-4c49-8701-dc9289afa4ad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264420 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264177 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264420 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264420 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264264 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264674 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264531 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264674 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264602 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264674 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264626 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.264842 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.264738 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.266622 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.266595 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.267027 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.267001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08841648-1c7b-4c49-8701-dc9289afa4ad-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.272735 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.272711 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnrhr\" (UniqueName: \"kubernetes.io/projected/08841648-1c7b-4c49-8701-dc9289afa4ad-kube-api-access-dnrhr\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.326426 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.326392 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:30.363031 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.362988 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:30.490355 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.490317 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x"] Jun 15 07:10:30.493228 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:10:30.493193 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499ddf64_dcf0_4692_946f_f4b8fcd4e1d8.slice/crio-e73ec7407e36c443937e473a463bccce21699263d19f949ba8fb165735cb58ec WatchSource:0}: Error finding container e73ec7407e36c443937e473a463bccce21699263d19f949ba8fb165735cb58ec: Status 404 returned error can't find the container with id e73ec7407e36c443937e473a463bccce21699263d19f949ba8fb165735cb58ec Jun 15 07:10:30.524609 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.524586 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Jun 15 07:10:30.534869 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:10:30.534838 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08841648_1c7b_4c49_8701_dc9289afa4ad.slice/crio-965422baf59b377b037f29580337b819645e62d42c1c264403f026e2d3c9d3d9 WatchSource:0}: Error finding container 965422baf59b377b037f29580337b819645e62d42c1c264403f026e2d3c9d3d9: Status 404 returned error can't find the container with id 965422baf59b377b037f29580337b819645e62d42c1c264403f026e2d3c9d3d9 Jun 15 07:10:30.929025 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.928979 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerStarted","Data":"a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b"} Jun 15 07:10:30.929517 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.929097 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerStarted","Data":"e73ec7407e36c443937e473a463bccce21699263d19f949ba8fb165735cb58ec"} Jun 15 07:10:30.930831 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.930786 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"08841648-1c7b-4c49-8701-dc9289afa4ad","Type":"ContainerStarted","Data":"f2d2764caafce993d43f48f503b09b8ff313f48d3018e2b77a1c7d158de03dae"} Jun 15 07:10:30.930939 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:30.930836 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"08841648-1c7b-4c49-8701-dc9289afa4ad","Type":"ContainerStarted","Data":"965422baf59b377b037f29580337b819645e62d42c1c264403f026e2d3c9d3d9"} Jun 15 07:10:31.937996 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:31.937962 2569 generic.go:356] "Generic (PLEG): container finished" podID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerID="a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b" exitCode=0 Jun 15 07:10:31.938347 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:31.938054 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerDied","Data":"a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b"} Jun 15 07:10:32.946829 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:32.946790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerStarted","Data":"00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25"} Jun 15 07:10:32.947238 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:32.946856 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:32.947238 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:32.946876 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerStarted","Data":"1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2"} Jun 15 07:10:32.970188 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:32.970122 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" podStartSLOduration=3.101952799 podStartE2EDuration="3.970099507s" totalImagesPullingTime="0s" totalInitContainerRuntime="868.146708ms" isStatefulPod=false podCreationTimestamp="2026-06-15 07:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:10:32.968953516 +0000 UTC m=+4911.314156659" watchObservedRunningTime="2026-06-15 07:10:32.970099507 +0000 UTC m=+4911.315302650" Jun 15 07:10:36.975965 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:36.975928 2569 generic.go:356] "Generic (PLEG): container finished" podID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerID="f2d2764caafce993d43f48f503b09b8ff313f48d3018e2b77a1c7d158de03dae" exitCode=0 Jun 15 07:10:36.976398 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:36.975998 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"08841648-1c7b-4c49-8701-dc9289afa4ad","Type":"ContainerDied","Data":"f2d2764caafce993d43f48f503b09b8ff313f48d3018e2b77a1c7d158de03dae"} Jun 15 07:10:37.984289 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:37.984246 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"08841648-1c7b-4c49-8701-dc9289afa4ad","Type":"ContainerStarted","Data":"b976d8f92193c471b074b78e8d73ad61b0d05ce5062218e1e765bc9a0d727583"} Jun 15 07:10:38.013262 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:38.013197 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=3.3187836920000002 podStartE2EDuration="9.013174977s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.694391285s" isStatefulPod=false podCreationTimestamp="2026-06-15 07:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:10:38.00922259 +0000 UTC m=+4916.354425767" watchObservedRunningTime="2026-06-15 07:10:38.013174977 +0000 UTC m=+4916.358378122" Jun 15 07:10:40.327157 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:40.327123 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:40.327654 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:40.327195 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:40.330134 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:40.330103 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:40.363860 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:40.363824 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:10:40.365420 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:40.365382 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:10:41.004067 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:41.004029 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:10:50.363404 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:50.363362 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:10:56.966938 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:10:56.966896 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:11:00.363876 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:00.363818 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:11:00.364347 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:00.364184 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:11:03.021482 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:03.021405 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:11:10.364460 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:10.364407 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:11:20.363442 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:20.363397 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:11:30.363507 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:30.363454 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:11:40.363481 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:40.363426 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:11:50.364148 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:11:50.364101 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:12:00.363749 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:12:00.363642 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:12:10.363431 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:12:10.363383 2569 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused" Jun 15 07:12:20.373349 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:12:20.373313 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:12:20.381300 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:12:20.381272 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:13:12.571965 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:13:12.571918 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:13:49.760804 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:13:49.760686 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:13:49.775866 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:13:49.775840 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:16:52.156383 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:16:52.156352 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:16:52.167131 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:16:52.167103 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:17:06.255415 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:06.255331 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:17:06.464947 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:06.464914 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:17:31.977499 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:31.977462 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:17:32.329991 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:32.329953 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:17:35.751036 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:35.751005 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:17:35.757051 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:35.757015 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x"] Jun 15 07:17:35.757351 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:35.757320 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="main" containerID="cri-o://1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2" gracePeriod=30 Jun 15 07:17:35.757466 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:35.757335 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="tokenizer" containerID="cri-o://00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25" gracePeriod=30 Jun 15 07:17:35.768095 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:35.768052 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Jun 15 07:17:35.817199 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:35.817162 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:17:36.387421 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:36.387390 2569 generic.go:356] "Generic (PLEG): container finished" podID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerID="1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2" exitCode=0 Jun 15 07:17:36.387610 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:36.387468 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerDied","Data":"1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2"} Jun 15 07:17:37.128164 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.128134 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:17:37.219276 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219183 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tls-certs\") pod \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " Jun 15 07:17:37.219454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219277 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmtr2\" (UniqueName: \"kubernetes.io/projected/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kube-api-access-mmtr2\") pod \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " Jun 15 07:17:37.219454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219311 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kserve-provision-location\") pod \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " Jun 15 07:17:37.219454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219352 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-uds\") pod \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " Jun 15 07:17:37.219454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219380 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-tmp\") pod \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " Jun 15 07:17:37.219454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219447 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-cache\") pod \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\" (UID: \"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8\") " Jun 15 07:17:37.219740 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219685 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" (UID: "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:37.219801 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219754 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" (UID: "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:37.219801 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219782 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" (UID: "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:37.219970 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219953 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:37.220017 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219975 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:37.220017 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.219989 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:37.220089 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.220035 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" (UID: "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:37.221516 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.221487 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" (UID: "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 07:17:37.221644 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.221611 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kube-api-access-mmtr2" (OuterVolumeSpecName: "kube-api-access-mmtr2") pod "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" (UID: "499ddf64-dcf0-4692-946f-f4b8fcd4e1d8"). InnerVolumeSpecName "kube-api-access-mmtr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 07:17:37.321246 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.321197 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:37.321410 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.321262 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmtr2\" (UniqueName: \"kubernetes.io/projected/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kube-api-access-mmtr2\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:37.321410 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.321274 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:37.394354 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.394319 2569 generic.go:356] "Generic (PLEG): container finished" podID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerID="00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25" exitCode=0 Jun 15 07:17:37.394560 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.394399 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" Jun 15 07:17:37.394560 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.394409 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerDied","Data":"00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25"} Jun 15 07:17:37.394560 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.394504 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x" event={"ID":"499ddf64-dcf0-4692-946f-f4b8fcd4e1d8","Type":"ContainerDied","Data":"e73ec7407e36c443937e473a463bccce21699263d19f949ba8fb165735cb58ec"} Jun 15 07:17:37.394560 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.394551 2569 scope.go:117] "RemoveContainer" containerID="00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25" Jun 15 07:17:37.404175 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.404145 2569 scope.go:117] "RemoveContainer" containerID="1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2" Jun 15 07:17:37.414844 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.414819 2569 scope.go:117] "RemoveContainer" containerID="a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b" Jun 15 07:17:37.419833 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.419806 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x"] Jun 15 07:17:37.423153 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.423124 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schegs79x"] Jun 15 07:17:37.425766 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.425749 2569 scope.go:117] "RemoveContainer" containerID="00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25" Jun 15 07:17:37.426057 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:17:37.426038 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25\": container with ID starting with 00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25 not found: ID does not exist" containerID="00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25" Jun 15 07:17:37.426115 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.426068 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25"} err="failed to get container status \"00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25\": rpc error: code = NotFound desc = could not find container \"00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25\": container with ID starting with 00ca731ce10d5c138e15e8d473afb5d04a2563fc5d1f6d46d2405d526da7ff25 not found: ID does not exist" Jun 15 07:17:37.426115 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.426087 2569 scope.go:117] "RemoveContainer" containerID="1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2" Jun 15 07:17:37.426293 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:17:37.426276 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2\": container with ID starting with 1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2 not found: ID does not exist" containerID="1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2" Jun 15 07:17:37.426336 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.426298 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2"} err="failed to get container status \"1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2\": rpc error: code = NotFound desc = could not find container \"1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2\": container with ID starting with 1558f29c3c114a9252f220d4a29e07888bf8152eff6591788b0405f69f8daab2 not found: ID does not exist" Jun 15 07:17:37.426336 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.426312 2569 scope.go:117] "RemoveContainer" containerID="a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b" Jun 15 07:17:37.426512 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:17:37.426496 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b\": container with ID starting with a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b not found: ID does not exist" containerID="a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b" Jun 15 07:17:37.426571 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:37.426516 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b"} err="failed to get container status \"a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b\": rpc error: code = NotFound desc = could not find container \"a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b\": container with ID starting with a28264e848842115c26d9f8a71d53fada502bb5270302603db4b8ee72f277d6b not found: ID does not exist" Jun 15 07:17:38.057735 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:38.057697 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" path="/var/lib/kubelet/pods/499ddf64-dcf0-4692-946f-f4b8fcd4e1d8/volumes" Jun 15 07:17:50.806206 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:50.806154 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" containerID="cri-o://b976d8f92193c471b074b78e8d73ad61b0d05ce5062218e1e765bc9a0d727583" gracePeriod=45 Jun 15 07:17:51.478741 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.478709 2569 generic.go:356] "Generic (PLEG): container finished" podID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerID="b976d8f92193c471b074b78e8d73ad61b0d05ce5062218e1e765bc9a0d727583" exitCode=0 Jun 15 07:17:51.478915 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.478779 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"08841648-1c7b-4c49-8701-dc9289afa4ad","Type":"ContainerDied","Data":"b976d8f92193c471b074b78e8d73ad61b0d05ce5062218e1e765bc9a0d727583"} Jun 15 07:17:51.573397 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.573372 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:17:51.674407 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674313 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-home\") pod \"08841648-1c7b-4c49-8701-dc9289afa4ad\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " Jun 15 07:17:51.674407 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674390 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-kserve-provision-location\") pod \"08841648-1c7b-4c49-8701-dc9289afa4ad\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " Jun 15 07:17:51.674667 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674431 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08841648-1c7b-4c49-8701-dc9289afa4ad-tls-certs\") pod \"08841648-1c7b-4c49-8701-dc9289afa4ad\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " Jun 15 07:17:51.674667 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674464 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-model-cache\") pod \"08841648-1c7b-4c49-8701-dc9289afa4ad\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " Jun 15 07:17:51.674667 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674644 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-dshm\") pod \"08841648-1c7b-4c49-8701-dc9289afa4ad\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " Jun 15 07:17:51.674837 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674693 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnrhr\" (UniqueName: \"kubernetes.io/projected/08841648-1c7b-4c49-8701-dc9289afa4ad-kube-api-access-dnrhr\") pod \"08841648-1c7b-4c49-8701-dc9289afa4ad\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " Jun 15 07:17:51.674837 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674751 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-tmp-dir\") pod \"08841648-1c7b-4c49-8701-dc9289afa4ad\" (UID: \"08841648-1c7b-4c49-8701-dc9289afa4ad\") " Jun 15 07:17:51.674947 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.674860 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "08841648-1c7b-4c49-8701-dc9289afa4ad" (UID: "08841648-1c7b-4c49-8701-dc9289afa4ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:51.675173 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.675147 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-home" (OuterVolumeSpecName: "home") pod "08841648-1c7b-4c49-8701-dc9289afa4ad" (UID: "08841648-1c7b-4c49-8701-dc9289afa4ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:51.675335 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.675201 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-model-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:51.676963 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.676932 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-dshm" (OuterVolumeSpecName: "dshm") pod "08841648-1c7b-4c49-8701-dc9289afa4ad" (UID: "08841648-1c7b-4c49-8701-dc9289afa4ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:51.677174 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.677142 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08841648-1c7b-4c49-8701-dc9289afa4ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08841648-1c7b-4c49-8701-dc9289afa4ad" (UID: "08841648-1c7b-4c49-8701-dc9289afa4ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 07:17:51.677257 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.677184 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08841648-1c7b-4c49-8701-dc9289afa4ad-kube-api-access-dnrhr" (OuterVolumeSpecName: "kube-api-access-dnrhr") pod "08841648-1c7b-4c49-8701-dc9289afa4ad" (UID: "08841648-1c7b-4c49-8701-dc9289afa4ad"). InnerVolumeSpecName "kube-api-access-dnrhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 07:17:51.688746 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.688718 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "08841648-1c7b-4c49-8701-dc9289afa4ad" (UID: "08841648-1c7b-4c49-8701-dc9289afa4ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:51.719720 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.719668 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "08841648-1c7b-4c49-8701-dc9289afa4ad" (UID: "08841648-1c7b-4c49-8701-dc9289afa4ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:17:51.776920 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.776882 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-dshm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:51.776920 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.776915 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnrhr\" (UniqueName: \"kubernetes.io/projected/08841648-1c7b-4c49-8701-dc9289afa4ad-kube-api-access-dnrhr\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:51.776920 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.776927 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-tmp-dir\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:51.777181 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.776935 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-home\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:51.777181 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.776944 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/08841648-1c7b-4c49-8701-dc9289afa4ad-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:51.777181 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:51.776953 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08841648-1c7b-4c49-8701-dc9289afa4ad-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:17:52.486099 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:52.486018 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" Jun 15 07:17:52.486099 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:52.486047 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"08841648-1c7b-4c49-8701-dc9289afa4ad","Type":"ContainerDied","Data":"965422baf59b377b037f29580337b819645e62d42c1c264403f026e2d3c9d3d9"} Jun 15 07:17:52.486620 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:52.486111 2569 scope.go:117] "RemoveContainer" containerID="b976d8f92193c471b074b78e8d73ad61b0d05ce5062218e1e765bc9a0d727583" Jun 15 07:17:52.495510 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:52.495470 2569 scope.go:117] "RemoveContainer" containerID="f2d2764caafce993d43f48f503b09b8ff313f48d3018e2b77a1c7d158de03dae" Jun 15 07:17:52.507355 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:52.507324 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Jun 15 07:17:52.512861 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:52.512825 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"] Jun 15 07:17:54.057374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:17:54.057333 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" path="/var/lib/kubelet/pods/08841648-1c7b-4c49-8701-dc9289afa4ad/volumes" Jun 15 07:18:04.792166 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:04.792112 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:18:25.287125 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.287086 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:18:25.532523 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.532474 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z"] Jun 15 07:18:25.533359 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533333 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="storage-initializer" Jun 15 07:18:25.533518 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533486 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="storage-initializer" Jun 15 07:18:25.533518 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533516 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="main" Jun 15 07:18:25.533724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533526 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="main" Jun 15 07:18:25.533724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533553 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="tokenizer" Jun 15 07:18:25.533724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533563 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="tokenizer" Jun 15 07:18:25.533724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533582 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" Jun 15 07:18:25.533724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533591 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" Jun 15 07:18:25.533724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533611 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="storage-initializer" Jun 15 07:18:25.533724 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533621 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="storage-initializer" Jun 15 07:18:25.534094 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533750 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="08841648-1c7b-4c49-8701-dc9289afa4ad" containerName="main" Jun 15 07:18:25.534094 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533765 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="tokenizer" Jun 15 07:18:25.534094 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.533778 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="499ddf64-dcf0-4692-946f-f4b8fcd4e1d8" containerName="main" Jun 15 07:18:25.538707 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.538632 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.542800 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.542774 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-8rmss\"" Jun 15 07:18:25.543290 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.543270 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\"" Jun 15 07:18:25.547283 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.547256 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z"] Jun 15 07:18:25.616031 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.615986 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.616031 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.616034 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.616294 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.616069 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.616294 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.616096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbsx\" (UniqueName: \"kubernetes.io/projected/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-kube-api-access-8bbsx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.616294 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.616119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.717177 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.717131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.717384 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.717202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.717384 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.717250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.717384 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.717280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbsx\" (UniqueName: \"kubernetes.io/projected/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-kube-api-access-8bbsx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.717384 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.717309 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.717873 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.717846 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.717960 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.717872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-tmp\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.718247 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.718216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-cache\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.720214 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.720193 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.726772 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.726751 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbsx\" (UniqueName: \"kubernetes.io/projected/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-kube-api-access-8bbsx\") pod \"scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.854119 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.854026 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:25.995598 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:25.995572 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z"] Jun 15 07:18:25.997751 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:18:25.997719 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b59ab7_3e90_45aa_8d72_7cd5293d7f98.slice/crio-6b0a9916bcc0b3707b7daf17b453c0235fbe2d3315ca2f3582361f5052ffe24c WatchSource:0}: Error finding container 6b0a9916bcc0b3707b7daf17b453c0235fbe2d3315ca2f3582361f5052ffe24c: Status 404 returned error can't find the container with id 6b0a9916bcc0b3707b7daf17b453c0235fbe2d3315ca2f3582361f5052ffe24c Jun 15 07:18:26.370687 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:26.370606 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:18:26.672376 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:26.672289 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" event={"ID":"99b59ab7-3e90-45aa-8d72-7cd5293d7f98","Type":"ContainerStarted","Data":"c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67"} Jun 15 07:18:26.672376 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:26.672351 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" event={"ID":"99b59ab7-3e90-45aa-8d72-7cd5293d7f98","Type":"ContainerStarted","Data":"20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb"} Jun 15 07:18:26.672625 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:26.672379 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" event={"ID":"99b59ab7-3e90-45aa-8d72-7cd5293d7f98","Type":"ContainerStarted","Data":"6b0a9916bcc0b3707b7daf17b453c0235fbe2d3315ca2f3582361f5052ffe24c"} Jun 15 07:18:26.672625 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:26.672500 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:26.694112 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:26.694052 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" podStartSLOduration=1.694034448 podStartE2EDuration="1.694034448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 07:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:18:26.691850391 +0000 UTC m=+5385.037053552" watchObservedRunningTime="2026-06-15 07:18:26.694034448 +0000 UTC m=+5385.039237589" Jun 15 07:18:35.855081 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:35.854995 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:35.855081 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:35.855060 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:35.857849 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:35.857819 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:36.730174 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:36.730138 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:44.779607 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:44.779568 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:18:49.801838 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:49.801807 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:18:49.821588 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:49.821563 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:18:57.248148 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:57.248095 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:18:57.735934 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:57.735902 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:18:58.690816 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:58.690780 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z"] Jun 15 07:18:58.691230 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:58.691039 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="main" containerID="cri-o://20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb" gracePeriod=30 Jun 15 07:18:58.691230 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:58.691064 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="tokenizer" containerID="cri-o://c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67" gracePeriod=30 Jun 15 07:18:58.692128 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:58.692048 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:18:58.753221 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:58.753180 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:18:58.850831 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:58.850802 2569 generic.go:356] "Generic (PLEG): container finished" podID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerID="20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb" exitCode=0 Jun 15 07:18:58.851002 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:18:58.850872 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" event={"ID":"99b59ab7-3e90-45aa-8d72-7cd5293d7f98","Type":"ContainerDied","Data":"20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb"} Jun 15 07:19:00.061316 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.061281 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:19:00.175958 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.175869 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-tmp\") pod \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " Jun 15 07:19:00.175958 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.175943 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-cache\") pod \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " Jun 15 07:19:00.176188 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176048 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-uds\") pod \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " Jun 15 07:19:00.176188 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176083 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tls-certs\") pod \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " Jun 15 07:19:00.176188 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176107 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bbsx\" (UniqueName: \"kubernetes.io/projected/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-kube-api-access-8bbsx\") pod \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\" (UID: \"99b59ab7-3e90-45aa-8d72-7cd5293d7f98\") " Jun 15 07:19:00.176360 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176202 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "99b59ab7-3e90-45aa-8d72-7cd5293d7f98" (UID: "99b59ab7-3e90-45aa-8d72-7cd5293d7f98"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:19:00.176360 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176296 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "99b59ab7-3e90-45aa-8d72-7cd5293d7f98" (UID: "99b59ab7-3e90-45aa-8d72-7cd5293d7f98"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:19:00.176360 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176332 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "99b59ab7-3e90-45aa-8d72-7cd5293d7f98" (UID: "99b59ab7-3e90-45aa-8d72-7cd5293d7f98"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:19:00.176950 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176925 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:19:00.176950 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176951 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:19:00.177121 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.176965 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:19:00.178358 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.178333 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99b59ab7-3e90-45aa-8d72-7cd5293d7f98" (UID: "99b59ab7-3e90-45aa-8d72-7cd5293d7f98"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 07:19:00.178739 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.178720 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-kube-api-access-8bbsx" (OuterVolumeSpecName: "kube-api-access-8bbsx") pod "99b59ab7-3e90-45aa-8d72-7cd5293d7f98" (UID: "99b59ab7-3e90-45aa-8d72-7cd5293d7f98"). InnerVolumeSpecName "kube-api-access-8bbsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 07:19:00.277864 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.277828 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:19:00.277864 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.277857 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbsx\" (UniqueName: \"kubernetes.io/projected/99b59ab7-3e90-45aa-8d72-7cd5293d7f98-kube-api-access-8bbsx\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:19:00.863778 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.863741 2569 generic.go:356] "Generic (PLEG): container finished" podID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerID="c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67" exitCode=0 Jun 15 07:19:00.863974 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.863809 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" Jun 15 07:19:00.863974 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.863825 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" event={"ID":"99b59ab7-3e90-45aa-8d72-7cd5293d7f98","Type":"ContainerDied","Data":"c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67"} Jun 15 07:19:00.863974 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.863880 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z" event={"ID":"99b59ab7-3e90-45aa-8d72-7cd5293d7f98","Type":"ContainerDied","Data":"6b0a9916bcc0b3707b7daf17b453c0235fbe2d3315ca2f3582361f5052ffe24c"} Jun 15 07:19:00.863974 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.863905 2569 scope.go:117] "RemoveContainer" containerID="c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67" Jun 15 07:19:00.874039 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.874016 2569 scope.go:117] "RemoveContainer" containerID="20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb" Jun 15 07:19:00.884042 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.884016 2569 scope.go:117] "RemoveContainer" containerID="c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67" Jun 15 07:19:00.884328 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:19:00.884305 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67\": container with ID starting with c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67 not found: ID does not exist" containerID="c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67" Jun 15 07:19:00.884412 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.884342 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67"} err="failed to get container status \"c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67\": rpc error: code = NotFound desc = could not find container \"c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67\": container with ID starting with c6437f167cd426c8b642521c82a61a843fa100c523594f256b0c17c223c6ee67 not found: ID does not exist" Jun 15 07:19:00.884412 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.884365 2569 scope.go:117] "RemoveContainer" containerID="20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb" Jun 15 07:19:00.884619 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:19:00.884597 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb\": container with ID starting with 20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb not found: ID does not exist" containerID="20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb" Jun 15 07:19:00.884663 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.884626 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb"} err="failed to get container status \"20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb\": rpc error: code = NotFound desc = could not find container \"20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb\": container with ID starting with 20d3e20104102a84d4f340131bd9061a086ebe5201f304d8b5598bfbae7b23fb not found: ID does not exist" Jun 15 07:19:00.890682 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.890659 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z"] Jun 15 07:19:00.898066 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:00.898043 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-5484fkg45z"] Jun 15 07:19:02.057400 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:02.057369 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" path="/var/lib/kubelet/pods/99b59ab7-3e90-45aa-8d72-7cd5293d7f98/volumes" Jun 15 07:19:38.673000 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.672965 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:19:38.864310 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.864274 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf"] Jun 15 07:19:38.864825 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.864803 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="main" Jun 15 07:19:38.864825 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.864824 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="main" Jun 15 07:19:38.864959 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.864837 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="tokenizer" Jun 15 07:19:38.864959 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.864843 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="tokenizer" Jun 15 07:19:38.864959 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.864915 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="tokenizer" Jun 15 07:19:38.864959 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.864925 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b59ab7-3e90-45aa-8d72-7cd5293d7f98" containerName="main" Jun 15 07:19:38.868068 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.868044 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:38.870330 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.870306 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\"" Jun 15 07:19:38.871020 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.871001 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-58hpl\"" Jun 15 07:19:38.878978 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.878957 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf"] Jun 15 07:19:38.887533 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.887505 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:19:38.975454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.975362 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-tmp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:38.975454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.975428 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-cache\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:38.975678 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.975490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdqpm\" (UniqueName: \"kubernetes.io/projected/128187ae-7990-4752-bd6b-a10912418eda-kube-api-access-rdqpm\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:38.975678 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.975611 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128187ae-7990-4752-bd6b-a10912418eda-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:38.975761 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:38.975680 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076416 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076383 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-tmp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076646 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076448 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-cache\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076646 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076468 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdqpm\" (UniqueName: \"kubernetes.io/projected/128187ae-7990-4752-bd6b-a10912418eda-kube-api-access-rdqpm\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076646 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076513 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128187ae-7990-4752-bd6b-a10912418eda-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076646 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076901 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-tmp\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076971 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076905 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-cache\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.076971 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.076927 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.079319 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.079281 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128187ae-7990-4752-bd6b-a10912418eda-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.085880 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.085846 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdqpm\" (UniqueName: \"kubernetes.io/projected/128187ae-7990-4752-bd6b-a10912418eda-kube-api-access-rdqpm\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.178718 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.178677 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:39.327587 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:39.327561 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf"] Jun 15 07:19:39.329841 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:19:39.329813 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128187ae_7990_4752_bd6b_a10912418eda.slice/crio-488b58538131ea7d30d79ff778c7ff17bd405dcdf712934ed51bd2fa1e608f0e WatchSource:0}: Error finding container 488b58538131ea7d30d79ff778c7ff17bd405dcdf712934ed51bd2fa1e608f0e: Status 404 returned error can't find the container with id 488b58538131ea7d30d79ff778c7ff17bd405dcdf712934ed51bd2fa1e608f0e Jun 15 07:19:40.073526 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:40.073488 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" event={"ID":"128187ae-7990-4752-bd6b-a10912418eda","Type":"ContainerStarted","Data":"0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2"} Jun 15 07:19:40.073934 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:40.073567 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" event={"ID":"128187ae-7990-4752-bd6b-a10912418eda","Type":"ContainerStarted","Data":"483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660"} Jun 15 07:19:40.073934 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:40.073598 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:40.073934 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:40.073614 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" event={"ID":"128187ae-7990-4752-bd6b-a10912418eda","Type":"ContainerStarted","Data":"488b58538131ea7d30d79ff778c7ff17bd405dcdf712934ed51bd2fa1e608f0e"} Jun 15 07:19:40.101760 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:40.101704 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" podStartSLOduration=2.10168675 podStartE2EDuration="2.10168675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 07:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:19:40.099741065 +0000 UTC m=+5458.444944243" watchObservedRunningTime="2026-06-15 07:19:40.10168675 +0000 UTC m=+5458.446889891" Jun 15 07:19:49.179627 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:49.179585 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:49.180199 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:49.179660 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:49.182261 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:49.182237 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:50.130292 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:50.130253 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:19:53.405667 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:53.405634 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:19:55.940872 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:55.940834 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:19:59.437018 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:59.436980 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:19:59.943870 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:19:59.943835 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:20:00.043491 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:00.043450 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:20:11.136580 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:11.136484 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:20:18.091336 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.091269 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:20:18.273617 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.273566 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8"] Jun 15 07:20:18.282627 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.282592 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.286535 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.286508 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-jwlwx\"" Jun 15 07:20:18.286800 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.286689 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\"" Jun 15 07:20:18.292141 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.292111 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8"] Jun 15 07:20:18.308117 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.308068 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:20:18.389802 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.389700 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.389802 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.389746 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9n9z\" (UniqueName: \"kubernetes.io/projected/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kube-api-access-w9n9z\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.390039 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.389819 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.390039 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.389893 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.390039 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.389948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.390186 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.390076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491306 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491268 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491306 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491579 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491352 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491579 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491579 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491397 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9n9z\" (UniqueName: \"kubernetes.io/projected/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kube-api-access-w9n9z\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491579 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491431 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491808 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491764 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491874 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491801 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491874 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491855 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.491874 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.491868 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.493967 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.493943 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.501587 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.501563 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9n9z\" (UniqueName: \"kubernetes.io/projected/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kube-api-access-w9n9z\") pod \"stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.596646 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.596609 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:18.762092 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.762061 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8"] Jun 15 07:20:18.764822 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:20:18.764789 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c8253a_1fd5_4730_a4b4_1a0e6bfc1e63.slice/crio-0f9a9195bc137c65ac67084b51d53dbcdb145a89fc371a3f03acaf830a57627e WatchSource:0}: Error finding container 0f9a9195bc137c65ac67084b51d53dbcdb145a89fc371a3f03acaf830a57627e: Status 404 returned error can't find the container with id 0f9a9195bc137c65ac67084b51d53dbcdb145a89fc371a3f03acaf830a57627e Jun 15 07:20:18.769609 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:18.769582 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jun 15 07:20:19.297939 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:19.297901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerStarted","Data":"9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f"} Jun 15 07:20:19.298361 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:19.297964 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerStarted","Data":"0f9a9195bc137c65ac67084b51d53dbcdb145a89fc371a3f03acaf830a57627e"} Jun 15 07:20:20.305229 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:20.305194 2569 generic.go:356] "Generic (PLEG): container finished" podID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerID="9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f" exitCode=0 Jun 15 07:20:20.305841 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:20.305257 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerDied","Data":"9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f"} Jun 15 07:20:21.315288 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:21.315232 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerStarted","Data":"20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7"} Jun 15 07:20:21.315817 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:21.315349 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerStarted","Data":"31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0"} Jun 15 07:20:21.315817 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:21.315384 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:21.340338 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:21.340272 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" podStartSLOduration=2.517274273 podStartE2EDuration="3.340249525s" totalImagesPullingTime="0s" totalInitContainerRuntime="822.975252ms" isStatefulPod=false podCreationTimestamp="2026-06-15 07:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:20:21.337001374 +0000 UTC m=+5499.682204514" watchObservedRunningTime="2026-06-15 07:20:21.340249525 +0000 UTC m=+5499.685452668" Jun 15 07:20:28.597710 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:28.597662 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:28.598223 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:28.597738 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:28.600484 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:28.600455 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:29.384460 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:29.384416 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:47.141016 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:47.140980 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:20:50.385425 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:50.385396 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:20:50.769211 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:50.769124 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:20:50.771194 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:50.771165 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf"] Jun 15 07:20:50.771481 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:50.771453 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="main" containerID="cri-o://483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660" gracePeriod=30 Jun 15 07:20:50.771726 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:50.771564 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="tokenizer" containerID="cri-o://0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2" gracePeriod=30 Jun 15 07:20:50.829084 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:50.829011 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:20:51.135803 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:20:51.135766 2569 logging.go:55] [core] [Channel #5225 SubChannel #5226]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.55:9003", ServerName: "10.134.0.55:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.55:9003: connect: connection refused" Jun 15 07:20:51.500969 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:51.500881 2569 generic.go:356] "Generic (PLEG): container finished" podID="128187ae-7990-4752-bd6b-a10912418eda" containerID="483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660" exitCode=0 Jun 15 07:20:51.501416 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:51.500963 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" event={"ID":"128187ae-7990-4752-bd6b-a10912418eda","Type":"ContainerDied","Data":"483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660"} Jun 15 07:20:52.136151 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.136106 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.55:9003\" within 1s: context deadline exceeded" Jun 15 07:20:52.148167 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.148109 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:20:52.247936 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.247899 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdqpm\" (UniqueName: \"kubernetes.io/projected/128187ae-7990-4752-bd6b-a10912418eda-kube-api-access-rdqpm\") pod \"128187ae-7990-4752-bd6b-a10912418eda\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " Jun 15 07:20:52.248121 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.247981 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128187ae-7990-4752-bd6b-a10912418eda-tls-certs\") pod \"128187ae-7990-4752-bd6b-a10912418eda\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " Jun 15 07:20:52.248121 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.248047 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-uds\") pod \"128187ae-7990-4752-bd6b-a10912418eda\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " Jun 15 07:20:52.248121 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.248084 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-tmp\") pod \"128187ae-7990-4752-bd6b-a10912418eda\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " Jun 15 07:20:52.248121 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.248101 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-cache\") pod \"128187ae-7990-4752-bd6b-a10912418eda\" (UID: \"128187ae-7990-4752-bd6b-a10912418eda\") " Jun 15 07:20:52.248455 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.248412 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "128187ae-7990-4752-bd6b-a10912418eda" (UID: "128187ae-7990-4752-bd6b-a10912418eda"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:20:52.248455 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.248439 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "128187ae-7990-4752-bd6b-a10912418eda" (UID: "128187ae-7990-4752-bd6b-a10912418eda"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:20:52.248688 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.248431 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "128187ae-7990-4752-bd6b-a10912418eda" (UID: "128187ae-7990-4752-bd6b-a10912418eda"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:20:52.250141 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.250123 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128187ae-7990-4752-bd6b-a10912418eda-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "128187ae-7990-4752-bd6b-a10912418eda" (UID: "128187ae-7990-4752-bd6b-a10912418eda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 07:20:52.250314 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.250294 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128187ae-7990-4752-bd6b-a10912418eda-kube-api-access-rdqpm" (OuterVolumeSpecName: "kube-api-access-rdqpm") pod "128187ae-7990-4752-bd6b-a10912418eda" (UID: "128187ae-7990-4752-bd6b-a10912418eda"). InnerVolumeSpecName "kube-api-access-rdqpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 07:20:52.349900 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.349829 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:20:52.349900 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.349865 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:20:52.349900 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.349880 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdqpm\" (UniqueName: \"kubernetes.io/projected/128187ae-7990-4752-bd6b-a10912418eda-kube-api-access-rdqpm\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:20:52.349900 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.349896 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/128187ae-7990-4752-bd6b-a10912418eda-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:20:52.349900 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.349907 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/128187ae-7990-4752-bd6b-a10912418eda-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:20:52.508580 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.508476 2569 generic.go:356] "Generic (PLEG): container finished" podID="128187ae-7990-4752-bd6b-a10912418eda" containerID="0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2" exitCode=0 Jun 15 07:20:52.508580 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.508525 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" Jun 15 07:20:52.509112 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.508575 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" event={"ID":"128187ae-7990-4752-bd6b-a10912418eda","Type":"ContainerDied","Data":"0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2"} Jun 15 07:20:52.509112 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.508646 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf" event={"ID":"128187ae-7990-4752-bd6b-a10912418eda","Type":"ContainerDied","Data":"488b58538131ea7d30d79ff778c7ff17bd405dcdf712934ed51bd2fa1e608f0e"} Jun 15 07:20:52.509112 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.508680 2569 scope.go:117] "RemoveContainer" containerID="0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2" Jun 15 07:20:52.519454 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.519436 2569 scope.go:117] "RemoveContainer" containerID="483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660" Jun 15 07:20:52.527480 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.527463 2569 scope.go:117] "RemoveContainer" containerID="0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2" Jun 15 07:20:52.527786 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:20:52.527768 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2\": container with ID starting with 0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2 not found: ID does not exist" containerID="0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2" Jun 15 07:20:52.527863 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.527800 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2"} err="failed to get container status \"0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2\": rpc error: code = NotFound desc = could not find container \"0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2\": container with ID starting with 0b936b7f40402f9cb248319d2ddab5d616eb9b44977d72349ec22846937c34f2 not found: ID does not exist" Jun 15 07:20:52.527863 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.527818 2569 scope.go:117] "RemoveContainer" containerID="483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660" Jun 15 07:20:52.528048 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:20:52.528031 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660\": container with ID starting with 483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660 not found: ID does not exist" containerID="483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660" Jun 15 07:20:52.528090 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.528055 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660"} err="failed to get container status \"483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660\": rpc error: code = NotFound desc = could not find container \"483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660\": container with ID starting with 483b771a74d6215b6cc6e95dcee87454de15b3e62e9eb882ea569e668de38660 not found: ID does not exist" Jun 15 07:20:52.533908 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.533877 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf"] Jun 15 07:20:52.538907 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:52.538878 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schegrhrf"] Jun 15 07:20:54.057170 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:20:54.057129 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="128187ae-7990-4752-bd6b-a10912418eda" path="/var/lib/kubelet/pods/128187ae-7990-4752-bd6b-a10912418eda/volumes" Jun 15 07:21:20.498286 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:21:20.498253 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:21:20.768170 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:21:20.768067 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:21:36.849628 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:21:36.849529 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:21:53.501910 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:21:53.501873 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:21:53.575525 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:21:53.575491 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:22:49.315517 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:49.315481 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8"] Jun 15 07:22:49.316144 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:49.315826 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="main" containerID="cri-o://31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0" gracePeriod=30 Jun 15 07:22:49.316144 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:49.315861 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="tokenizer" containerID="cri-o://20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7" gracePeriod=30 Jun 15 07:22:49.320218 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:49.320165 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:22:49.377116 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:49.377078 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:22:49.379960 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:49.379928 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="tokenizer" probeResult="failure" output="Get \"http://10.134.0.56:8082/healthz\": dial tcp 10.134.0.56:8082: connect: connection refused" Jun 15 07:22:50.145754 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.145717 2569 generic.go:356] "Generic (PLEG): container finished" podID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerID="31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0" exitCode=0 Jun 15 07:22:50.145968 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.145776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerDied","Data":"31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0"} Jun 15 07:22:50.384719 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:22:50.384686 2569 logging.go:55] [core] [Channel #5440 SubChannel #5441]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.56:9003", ServerName: "10.134.0.56:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.56:9003: connect: connection refused" Jun 15 07:22:50.681990 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.681959 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:22:50.785480 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785448 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9n9z\" (UniqueName: \"kubernetes.io/projected/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kube-api-access-w9n9z\") pod \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " Jun 15 07:22:50.785701 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785523 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-cache\") pod \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " Jun 15 07:22:50.785701 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785592 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kserve-provision-location\") pod \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " Jun 15 07:22:50.785701 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785619 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-tmp\") pod \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " Jun 15 07:22:50.785701 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785687 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tls-certs\") pod \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " Jun 15 07:22:50.785939 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785819 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-uds\") pod \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\" (UID: \"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63\") " Jun 15 07:22:50.786009 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785816 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" (UID: "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:22:50.786009 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.785985 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" (UID: "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:22:50.786113 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.786097 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" (UID: "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:22:50.786283 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.786264 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:22:50.786376 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.786289 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:22:50.786376 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.786304 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:22:50.786514 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.786493 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" (UID: "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:22:50.787733 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.787708 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kube-api-access-w9n9z" (OuterVolumeSpecName: "kube-api-access-w9n9z") pod "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" (UID: "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63"). InnerVolumeSpecName "kube-api-access-w9n9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 07:22:50.787823 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.787769 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" (UID: "f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 07:22:50.887788 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.887749 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9n9z\" (UniqueName: \"kubernetes.io/projected/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kube-api-access-w9n9z\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:22:50.887788 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.887785 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:22:50.888002 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:50.887800 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:22:51.152374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.152278 2569 generic.go:356] "Generic (PLEG): container finished" podID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerID="20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7" exitCode=0 Jun 15 07:22:51.152374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.152338 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerDied","Data":"20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7"} Jun 15 07:22:51.152636 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.152385 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" event={"ID":"f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63","Type":"ContainerDied","Data":"0f9a9195bc137c65ac67084b51d53dbcdb145a89fc371a3f03acaf830a57627e"} Jun 15 07:22:51.152636 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.152407 2569 scope.go:117] "RemoveContainer" containerID="20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7" Jun 15 07:22:51.152636 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.152340 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" Jun 15 07:22:51.162639 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.162618 2569 scope.go:117] "RemoveContainer" containerID="31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0" Jun 15 07:22:51.171992 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.171971 2569 scope.go:117] "RemoveContainer" containerID="9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f" Jun 15 07:22:51.178409 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.178380 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8"] Jun 15 07:22:51.181868 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.181847 2569 scope.go:117] "RemoveContainer" containerID="20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7" Jun 15 07:22:51.182283 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:22:51.182213 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7\": container with ID starting with 20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7 not found: ID does not exist" containerID="20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7" Jun 15 07:22:51.182283 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.182254 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7"} err="failed to get container status \"20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7\": rpc error: code = NotFound desc = could not find container \"20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7\": container with ID starting with 20f4d27c2db9667429ab4bf6b6d9577a2921017ae63b048447e9ecdcf85c9ae7 not found: ID does not exist" Jun 15 07:22:51.182283 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.182281 2569 scope.go:117] "RemoveContainer" containerID="31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0" Jun 15 07:22:51.182603 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:22:51.182582 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0\": container with ID starting with 31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0 not found: ID does not exist" containerID="31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0" Jun 15 07:22:51.182686 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.182613 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0"} err="failed to get container status \"31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0\": rpc error: code = NotFound desc = could not find container \"31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0\": container with ID starting with 31cc7a6591a74a39f1664a9efc2d0dbd4d1a717ebc9a2f42b864664d8a7af3d0 not found: ID does not exist" Jun 15 07:22:51.182686 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.182636 2569 scope.go:117] "RemoveContainer" containerID="9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f" Jun 15 07:22:51.182927 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:22:51.182904 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f\": container with ID starting with 9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f not found: ID does not exist" containerID="9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f" Jun 15 07:22:51.182981 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.182935 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f"} err="failed to get container status \"9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f\": rpc error: code = NotFound desc = could not find container \"9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f\": container with ID starting with 9f75272323e2a74a985b7a6f6a4819d64f080b21f20eda2e5d6b21b01268022f not found: ID does not exist" Jun 15 07:22:51.184133 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.184112 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8"] Jun 15 07:22:51.384770 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:51.384722 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-7f74f857ff-5fvn8" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.56:9003\" within 1s: context deadline exceeded" Jun 15 07:22:52.057900 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:22:52.057867 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" path="/var/lib/kubelet/pods/f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63/volumes" Jun 15 07:23:01.303133 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:23:01.303055 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:23:01.364488 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:23:01.364457 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:23:18.060486 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:23:18.060455 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:23:49.840249 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:23:49.840140 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:23:49.858684 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:23:49.858661 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:24:29.260141 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:24:29.260108 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:25:03.949866 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:25:03.949833 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:25:13.374232 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:25:13.374198 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:25:41.268150 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:25:41.268106 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:26:22.359433 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:26:22.359396 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:27:47.473251 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:27:47.473213 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:28:39.869698 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:39.869650 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:28:40.117728 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.117691 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv"] Jun 15 07:28:40.118229 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118209 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="main" Jun 15 07:28:40.118312 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118231 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="main" Jun 15 07:28:40.118312 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118278 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="tokenizer" Jun 15 07:28:40.118312 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118285 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="tokenizer" Jun 15 07:28:40.118312 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118292 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="tokenizer" Jun 15 07:28:40.118312 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118298 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="tokenizer" Jun 15 07:28:40.118312 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118305 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="main" Jun 15 07:28:40.118312 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118310 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="main" Jun 15 07:28:40.118750 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118321 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="storage-initializer" Jun 15 07:28:40.118750 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118328 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="storage-initializer" Jun 15 07:28:40.118750 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118406 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="tokenizer" Jun 15 07:28:40.118750 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118420 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2c8253a-1fd5-4730-a4b4-1a0e6bfc1e63" containerName="main" Jun 15 07:28:40.118750 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118428 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="main" Jun 15 07:28:40.118750 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.118435 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="128187ae-7990-4752-bd6b-a10912418eda" containerName="tokenizer" Jun 15 07:28:40.122038 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.121979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.124517 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.124493 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-p9hx2\"" Jun 15 07:28:40.124685 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.124520 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\"" Jun 15 07:28:40.133985 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.133958 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv"] Jun 15 07:28:40.150617 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.150581 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:28:40.250307 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.250275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-tmp\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.250482 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.250337 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.250482 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.250379 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-cache\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.250482 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.250401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtslv\" (UniqueName: \"kubernetes.io/projected/be319099-87c8-4296-b22c-cf490bb80894-kube-api-access-xtslv\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.250482 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.250433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.250642 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.250501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be319099-87c8-4296-b22c-cf490bb80894-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352038 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-tmp\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352239 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352239 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352089 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-cache\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352367 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352245 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtslv\" (UniqueName: \"kubernetes.io/projected/be319099-87c8-4296-b22c-cf490bb80894-kube-api-access-xtslv\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352367 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352367 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352361 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be319099-87c8-4296-b22c-cf490bb80894-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352532 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352513 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-tmp\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352611 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352556 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352654 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352609 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-cache\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.352714 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.352696 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.355102 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.355079 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be319099-87c8-4296-b22c-cf490bb80894-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.362861 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.362837 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtslv\" (UniqueName: \"kubernetes.io/projected/be319099-87c8-4296-b22c-cf490bb80894-kube-api-access-xtslv\") pod \"prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.434127 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.434028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:40.587674 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.587641 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv"] Jun 15 07:28:40.590790 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:28:40.590756 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe319099_87c8_4296_b22c_cf490bb80894.slice/crio-f3e2d57518560b050851ebed97128cdfd99dabcab92cb9c9206a3cbe38973a94 WatchSource:0}: Error finding container f3e2d57518560b050851ebed97128cdfd99dabcab92cb9c9206a3cbe38973a94: Status 404 returned error can't find the container with id f3e2d57518560b050851ebed97128cdfd99dabcab92cb9c9206a3cbe38973a94 Jun 15 07:28:40.595469 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:40.595450 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Jun 15 07:28:41.059246 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:41.059207 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerStarted","Data":"a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f"} Jun 15 07:28:41.059817 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:41.059264 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerStarted","Data":"f3e2d57518560b050851ebed97128cdfd99dabcab92cb9c9206a3cbe38973a94"} Jun 15 07:28:42.066841 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:42.066807 2569 generic.go:356] "Generic (PLEG): container finished" podID="be319099-87c8-4296-b22c-cf490bb80894" containerID="a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f" exitCode=0 Jun 15 07:28:42.067282 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:42.066897 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerDied","Data":"a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f"} Jun 15 07:28:43.082121 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:43.082081 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerStarted","Data":"b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13"} Jun 15 07:28:43.082603 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:43.082179 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerStarted","Data":"d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80"} Jun 15 07:28:43.082603 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:43.082231 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:43.108710 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:43.108655 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" podStartSLOduration=2.284295901 podStartE2EDuration="3.108639354s" totalImagesPullingTime="0s" totalInitContainerRuntime="824.343453ms" isStatefulPod=false podCreationTimestamp="2026-06-15 07:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:28:43.105965615 +0000 UTC m=+6001.451168787" watchObservedRunningTime="2026-06-15 07:28:43.108639354 +0000 UTC m=+6001.453842494" Jun 15 07:28:49.885259 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:49.885146 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:28:49.900561 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:49.900518 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:28:50.434781 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:50.434741 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:50.434998 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:50.434826 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:50.437910 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:50.437883 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:51.132476 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:51.132440 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:28:51.868527 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:28:51.868488 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:29:12.138140 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:29:12.138060 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:31:02.974439 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:02.974405 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv"] Jun 15 07:31:02.975021 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:02.974710 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="main" containerID="cri-o://d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80" gracePeriod=30 Jun 15 07:31:02.975021 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:02.974710 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="tokenizer" containerID="cri-o://b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13" gracePeriod=30 Jun 15 07:31:02.994818 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:02.994769 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:31:03.857520 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:03.857481 2569 generic.go:356] "Generic (PLEG): container finished" podID="be319099-87c8-4296-b22c-cf490bb80894" containerID="d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80" exitCode=0 Jun 15 07:31:03.857741 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:03.857535 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerDied","Data":"d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80"} Jun 15 07:31:04.029880 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.029841 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:31:04.360806 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.360731 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:31:04.522913 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.522883 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtslv\" (UniqueName: \"kubernetes.io/projected/be319099-87c8-4296-b22c-cf490bb80894-kube-api-access-xtslv\") pod \"be319099-87c8-4296-b22c-cf490bb80894\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " Jun 15 07:31:04.522913 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.522919 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-uds\") pod \"be319099-87c8-4296-b22c-cf490bb80894\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " Jun 15 07:31:04.523161 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.522962 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-kserve-provision-location\") pod \"be319099-87c8-4296-b22c-cf490bb80894\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " Jun 15 07:31:04.523161 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523028 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-tmp\") pod \"be319099-87c8-4296-b22c-cf490bb80894\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " Jun 15 07:31:04.523161 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523058 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be319099-87c8-4296-b22c-cf490bb80894-tls-certs\") pod \"be319099-87c8-4296-b22c-cf490bb80894\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " Jun 15 07:31:04.523161 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523098 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-cache\") pod \"be319099-87c8-4296-b22c-cf490bb80894\" (UID: \"be319099-87c8-4296-b22c-cf490bb80894\") " Jun 15 07:31:04.523382 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523214 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "be319099-87c8-4296-b22c-cf490bb80894" (UID: "be319099-87c8-4296-b22c-cf490bb80894"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:31:04.523439 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523391 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "be319099-87c8-4296-b22c-cf490bb80894" (UID: "be319099-87c8-4296-b22c-cf490bb80894"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:31:04.523497 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523431 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "be319099-87c8-4296-b22c-cf490bb80894" (UID: "be319099-87c8-4296-b22c-cf490bb80894"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:31:04.523578 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523499 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-tmp\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:31:04.523578 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523512 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-uds\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:31:04.523925 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.523900 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be319099-87c8-4296-b22c-cf490bb80894" (UID: "be319099-87c8-4296-b22c-cf490bb80894"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jun 15 07:31:04.525200 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.525179 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be319099-87c8-4296-b22c-cf490bb80894-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be319099-87c8-4296-b22c-cf490bb80894" (UID: "be319099-87c8-4296-b22c-cf490bb80894"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jun 15 07:31:04.525286 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.525265 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be319099-87c8-4296-b22c-cf490bb80894-kube-api-access-xtslv" (OuterVolumeSpecName: "kube-api-access-xtslv") pod "be319099-87c8-4296-b22c-cf490bb80894" (UID: "be319099-87c8-4296-b22c-cf490bb80894"). InnerVolumeSpecName "kube-api-access-xtslv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jun 15 07:31:04.624379 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.624287 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be319099-87c8-4296-b22c-cf490bb80894-tls-certs\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:31:04.624379 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.624317 2569 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-tokenizer-cache\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:31:04.624379 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.624329 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtslv\" (UniqueName: \"kubernetes.io/projected/be319099-87c8-4296-b22c-cf490bb80894-kube-api-access-xtslv\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:31:04.624379 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.624338 2569 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be319099-87c8-4296-b22c-cf490bb80894-kserve-provision-location\") on node \"ip-10-0-128-226.ec2.internal\" DevicePath \"\"" Jun 15 07:31:04.865493 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.865453 2569 generic.go:356] "Generic (PLEG): container finished" podID="be319099-87c8-4296-b22c-cf490bb80894" containerID="b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13" exitCode=0 Jun 15 07:31:04.865679 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.865503 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" Jun 15 07:31:04.865679 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.865519 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerDied","Data":"b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13"} Jun 15 07:31:04.865679 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.865598 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv" event={"ID":"be319099-87c8-4296-b22c-cf490bb80894","Type":"ContainerDied","Data":"f3e2d57518560b050851ebed97128cdfd99dabcab92cb9c9206a3cbe38973a94"} Jun 15 07:31:04.865679 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.865632 2569 scope.go:117] "RemoveContainer" containerID="b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13" Jun 15 07:31:04.877522 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.877497 2569 scope.go:117] "RemoveContainer" containerID="d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80" Jun 15 07:31:04.887238 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.887215 2569 scope.go:117] "RemoveContainer" containerID="a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f" Jun 15 07:31:04.895648 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.895598 2569 scope.go:117] "RemoveContainer" containerID="b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13" Jun 15 07:31:04.896263 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:31:04.896235 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13\": container with ID starting with b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13 not found: ID does not exist" containerID="b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13" Jun 15 07:31:04.896364 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.896273 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13"} err="failed to get container status \"b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13\": rpc error: code = NotFound desc = could not find container \"b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13\": container with ID starting with b0947eaa6a295186e5b06e27791f462f2e2957eadfec2a1be1a3ba6517445d13 not found: ID does not exist" Jun 15 07:31:04.896364 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.896298 2569 scope.go:117] "RemoveContainer" containerID="d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80" Jun 15 07:31:04.896674 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:31:04.896648 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80\": container with ID starting with d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80 not found: ID does not exist" containerID="d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80" Jun 15 07:31:04.896756 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.896684 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80"} err="failed to get container status \"d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80\": rpc error: code = NotFound desc = could not find container \"d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80\": container with ID starting with d0212821fff3507cfae7c834ea3d984ee8c6cd85664d1b98a0efb44f8f643c80 not found: ID does not exist" Jun 15 07:31:04.896756 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.896707 2569 scope.go:117] "RemoveContainer" containerID="a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f" Jun 15 07:31:04.897100 ip-10-0-128-226 kubenswrapper[2569]: E0615 07:31:04.897077 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f\": container with ID starting with a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f not found: ID does not exist" containerID="a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f" Jun 15 07:31:04.897186 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.897112 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f"} err="failed to get container status \"a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f\": rpc error: code = NotFound desc = could not find container \"a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f\": container with ID starting with a57de430e1f65366c3c8cc0a68fec63db03cbd58b67dc8f6c2bfc864c79f391f not found: ID does not exist" Jun 15 07:31:04.898701 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.898676 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv"] Jun 15 07:31:04.905349 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:04.905327 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-8578fc44f9-fs8xv"] Jun 15 07:31:06.057161 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:06.057126 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be319099-87c8-4296-b22c-cf490bb80894" path="/var/lib/kubelet/pods/be319099-87c8-4296-b22c-cf490bb80894/volumes" Jun 15 07:31:52.372013 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:52.371969 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:31:52.428273 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:31:52.428239 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:33:37.919801 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:37.919721 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:33:37.999753 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:37.999715 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:33:49.923925 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:49.923803 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:33:49.944806 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:49.944780 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rkdkp_4354a742-ec1d-4616-ad21-3f07511a6877/ovn-acl-logging/0.log" Jun 15 07:33:53.951048 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:53.951012 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:33:54.005220 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.005188 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:33:54.024049 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.024019 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:33:54.092630 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.092597 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:33:54.214020 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.213937 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:33:54.233334 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.233312 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:33:54.466379 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.466294 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:33:54.467010 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.466981 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:33:54.502171 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.502141 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:33:54.517781 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.517754 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:33:54.528119 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.528085 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:33:54.560349 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.560319 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:33:54.569755 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.569731 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:33:54.669709 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.669679 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:33:54.678155 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.678125 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:33:54.696787 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.696761 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:33:54.718143 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.718058 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:33:54.726966 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:54.726940 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:33:56.297567 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.297518 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:33:56.319940 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.319911 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:33:56.328182 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.328161 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:33:56.367092 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.367068 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:33:56.453534 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.453503 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:33:56.461372 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.461348 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:33:56.647926 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.647844 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:33:56.648885 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.648852 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:33:56.695205 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.695174 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:33:56.716897 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.716859 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:33:56.733140 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.733108 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:33:56.769157 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.769124 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:33:56.776966 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.776932 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:33:56.869273 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.869238 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:33:56.877941 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.877915 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:33:56.899618 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.899531 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:33:56.923682 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.923652 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:33:56.933095 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:56.933059 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:33:58.511171 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.511134 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:33:58.533985 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.533947 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:33:58.551938 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.551911 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:33:58.600403 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.600372 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:33:58.686835 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.686802 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:33:58.697811 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.697784 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:33:58.774663 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.774576 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:33:58.903952 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.903924 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:33:58.905311 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.905289 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:33:58.945618 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.945591 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:33:58.967596 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.967564 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:33:58.985404 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:58.985381 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:33:59.024253 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:59.024224 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:33:59.036178 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:59.036135 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:33:59.128411 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:59.128381 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:33:59.145723 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:59.145688 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:33:59.167703 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:59.167678 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:33:59.191654 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:59.191623 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:33:59.201956 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:33:59.201929 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:00.835867 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:00.835828 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:00.860365 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:00.860336 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:00.867757 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:00.867730 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:00.896130 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:00.896095 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:00.973932 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:00.973898 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:00.980499 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:00.980476 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:01.178356 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.178266 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:01.179438 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.179416 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:01.224200 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.224163 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:01.239752 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.239724 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:01.252688 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.252661 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:01.282123 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.282098 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:01.289629 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.289604 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:01.382126 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.382094 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:01.389981 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.389959 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:01.408175 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.408150 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:01.430195 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.430124 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:01.439254 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:01.439230 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:03.007608 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.007569 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:03.028839 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.028813 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:03.037578 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.037553 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:03.076332 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.076306 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:03.159180 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.159148 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:03.166703 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.166679 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:03.360606 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.360577 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:03.361642 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.361615 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:03.394712 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.394683 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:03.411537 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.411510 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:03.421661 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.421633 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:03.451015 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.450980 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:03.458449 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.458425 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:03.571716 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.571688 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:03.585230 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.585209 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:03.617353 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.617244 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:03.646561 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.646523 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:03.674170 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:03.674142 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:05.429975 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.429947 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:05.453486 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.453456 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:05.468082 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.468047 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:05.503942 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.503905 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:05.597995 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.597963 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:05.606255 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.606231 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:05.811186 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.811152 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:05.812406 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.812380 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:05.852389 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.852363 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:05.867984 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.867939 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:05.877923 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.877899 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:05.908973 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.908946 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:05.918695 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:05.918669 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:06.004283 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:06.004251 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:06.012247 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:06.012221 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:06.033447 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:06.033413 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:06.059233 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:06.059201 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:06.076841 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:06.076770 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:07.799742 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:07.799709 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:07.830434 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:07.830404 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:07.840865 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:07.840836 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:07.884999 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:07.884970 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:07.978321 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:07.978292 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:07.987104 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:07.987081 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:08.249342 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.249255 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:08.250383 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.250368 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:08.293179 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.293150 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:08.307705 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.307677 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:08.315186 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.315156 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:08.350198 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.350163 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:08.357935 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.357902 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:08.444771 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.444740 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:08.456375 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.456347 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:08.472579 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.472525 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:08.495496 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.495469 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:08.502602 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:08.502527 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:10.065690 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.065660 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:10.083599 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.083562 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:10.092081 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.092057 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:10.125043 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.125014 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:10.213848 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.213776 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:10.224887 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.224857 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:10.419391 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.419309 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:10.420482 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.420455 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:10.456607 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.456580 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:10.473395 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.473354 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:10.483842 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.483813 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:10.512202 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.512166 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:10.520347 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.520320 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:10.610875 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.610840 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:10.621776 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.621736 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:10.643560 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.643520 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:10.666895 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.666868 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:10.674666 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:10.674594 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:12.285753 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.285720 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:12.317348 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.317319 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:12.335593 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.335566 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:12.373838 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.373807 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:12.458367 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.458317 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:12.465715 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.465683 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:12.667863 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.667779 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:12.669013 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.668989 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:12.703339 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.703312 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:12.719647 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.719611 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:12.733763 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.733738 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:12.769604 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.769575 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:12.777051 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.777029 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:12.864513 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.864483 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:12.874879 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.874852 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:12.893035 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.893011 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:12.914083 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.914055 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:12.921672 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:12.921607 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:14.521168 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.521134 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:14.541994 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.541963 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:14.551388 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.551364 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:14.597507 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.597480 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:14.697559 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.697524 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:14.705743 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.705714 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:14.943316 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.943283 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:14.944439 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.944421 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:14.985088 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:14.985049 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:15.006221 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.006195 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:15.020936 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.020912 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:15.051704 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.051665 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:15.059260 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.059231 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:15.147491 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.147457 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:15.157181 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.157148 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:15.176087 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.176063 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:15.201618 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.201536 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:15.212796 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:15.212766 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:16.783364 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:16.783335 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:16.808102 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:16.808061 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:16.818205 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:16.818177 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:16.870747 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:16.870718 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:17.001266 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.001228 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:17.016508 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.016485 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:17.490811 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.490779 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:17.492192 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.492176 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:17.530813 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.530786 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:17.549792 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.549763 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:17.561428 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.561406 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:17.591081 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.591053 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:17.602443 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.602423 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:17.705178 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.705144 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:17.713648 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.713620 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:17.745460 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.745379 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:17.769605 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.769560 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:17.781132 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:17.781107 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:19.335585 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.335532 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:19.354606 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.354572 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:19.363024 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.362995 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:19.397302 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.397270 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:19.475533 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.475501 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:19.482750 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.482722 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:19.683254 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.683163 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:19.684578 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.684494 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:19.722200 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.722171 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:19.739195 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.739167 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:19.748347 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.748323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:19.775759 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.775732 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:19.783499 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.783470 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:19.874329 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.874297 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:19.881801 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.881775 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:19.905066 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.905039 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:19.947002 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.946928 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:19.973803 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:19.973777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:21.607778 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.607741 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:21.628500 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.628466 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:21.643699 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.643670 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:21.682021 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.681990 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:21.788040 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.788007 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:21.797413 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.797383 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:21.994487 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.994402 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:21.995686 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:21.995664 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:22.037251 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.037221 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:22.051114 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.051088 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:22.059774 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.059752 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:22.086064 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.086040 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:22.093920 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.093902 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:22.187167 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.187134 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:22.194800 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.194774 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:22.213249 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.213220 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:22.240398 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.240372 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:22.250253 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.250197 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:22.845166 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:22.845133 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:34:23.818342 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:23.818306 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:23.847058 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:23.847012 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:23.855051 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:23.855027 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:23.888606 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:23.888577 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:23.979906 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:23.979878 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:23.994340 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:23.994313 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:24.231321 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.231239 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:24.232493 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.232471 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:24.287999 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.287971 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:24.321329 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.321301 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:24.356024 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.355994 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:24.395379 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.395346 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:24.411319 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.411286 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:24.501145 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.501051 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:24.508950 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.508917 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:24.531249 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.531221 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:24.555980 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.555948 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:24.568952 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:24.568926 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:28.078471 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.078442 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:28.095973 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.095946 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:28.108243 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.108215 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:28.142491 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.142458 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:28.225506 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.225478 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:28.233993 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.233970 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:28.441715 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.441619 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:28.442851 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.442827 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:28.482799 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.482768 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:28.497533 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.497507 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:28.505648 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.505622 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:28.535142 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.535115 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:28.542893 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.542861 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:28.648013 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.647980 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:28.672898 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.672850 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:28.700204 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.700129 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:28.744274 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.744242 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:28.776662 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:28.776634 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:30.423590 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.423557 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:30.469862 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.469829 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:30.493508 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.493475 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:30.525613 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.525576 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:30.610022 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.609991 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:30.617218 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.617195 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:30.847780 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.847746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:30.848860 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.848842 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:30.889208 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.889173 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:30.904099 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.904070 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:30.914934 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.914910 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:30.942426 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.942392 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:30.950070 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:30.950046 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:31.048128 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:31.048097 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:31.056975 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:31.056949 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:31.075574 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:31.075526 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:31.094784 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:31.094757 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:31.102922 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:31.102862 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:32.642245 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.642211 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:32.664338 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.664305 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:32.671744 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.671718 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:32.703189 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.703163 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:32.780791 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.780760 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:32.788301 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.788273 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:32.984689 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.984597 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:32.986275 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:32.986253 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:33.029376 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.029349 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:33.043293 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.043264 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:33.050619 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.050592 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:33.080308 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.080270 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:33.087274 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.087246 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:33.167634 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.167597 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:33.175621 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.175596 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:33.192897 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.192861 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:33.225234 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.225205 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:33.232693 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:33.232663 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:34.776076 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:34.776048 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:34.798348 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:34.798319 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:34.805120 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:34.805097 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:34.833450 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:34.833414 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:34.919948 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:34.919919 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:34.935805 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:34.935777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:35.148100 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.148071 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:35.149445 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.149418 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:35.184426 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.184395 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:35.202476 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.202448 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:35.215324 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.215293 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:35.242226 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.242193 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:35.252706 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.252686 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:35.342220 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.342192 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:35.355810 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.355780 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:35.378992 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.378966 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:35.400488 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.400420 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:35.408871 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:35.408848 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:36.947170 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:36.947140 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:36.968390 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:36.968360 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:36.976556 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:36.976508 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:37.012572 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.012527 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:37.096423 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.096394 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:37.103904 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.103878 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:37.347384 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.347348 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:37.348745 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.348726 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:37.388004 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.387967 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:37.401182 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.401152 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:37.409049 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.409014 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:37.437234 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.437199 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:37.445476 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.445453 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:37.533070 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.533041 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:37.540564 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.540506 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:37.559518 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.559493 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:37.582866 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.582839 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:37.590403 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:37.590384 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:39.112133 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.112092 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/main/0.log" Jun 15 07:34:39.135749 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.135722 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/tokenizer/0.log" Jun 15 07:34:39.146641 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.146617 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_auth-enabled-test-kserve-router-scheduler-6c5d597fbb-nhwh9_9359d817-81c6-4fe2-80e0-76d36607616d/storage-initializer/0.log" Jun 15 07:34:39.177044 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.177019 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6dbc7ddb8d-c625w_2bcc7d6e-7945-4c51-910d-4a0ffe93df64/storage-initializer/0.log" Jun 15 07:34:39.256761 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.256717 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/main/0.log" Jun 15 07:34:39.263804 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.263778 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-766cc944c5-85gl8_8572c5e6-3714-4917-b782-31ec52c11134/storage-initializer/0.log" Jun 15 07:34:39.459083 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.458989 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/0.log" Jun 15 07:34:39.460648 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.460621 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-75dcfd69c9-dh6qf_e8405d0a-7af6-4f75-9d69-2a63b8579195/istio-proxy/1.log" Jun 15 07:34:39.496557 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.496522 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/main/0.log" Jun 15 07:34:39.512362 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.512333 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/llm-d-routing-sidecar/0.log" Jun 15 07:34:39.522282 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.522252 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-6f78896447-wshh4_5d2cc4b0-4097-4d57-9116-4a62c41cc76d/storage-initializer/0.log" Jun 15 07:34:39.552115 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.552083 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/main/0.log" Jun 15 07:34:39.559739 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.559712 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-5fc8578dd5-d6lhp_9c26092a-3ad8-4fc2-879a-4b96a47c9166/storage-initializer/0.log" Jun 15 07:34:39.647021 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.646977 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/main/0.log" Jun 15 07:34:39.654794 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.654773 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-578d595fc-gtvkx_c15f33fe-f482-409c-8011-41250b69694d/storage-initializer/0.log" Jun 15 07:34:39.672777 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.672753 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/main/0.log" Jun 15 07:34:39.692466 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.692438 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/tokenizer/0.log" Jun 15 07:34:39.699900 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:39.699877 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-router-scheduler-7d4868d689-h4c76_c837db67-d0c8-447e-87c9-5dec6015f51f/storage-initializer/0.log" Jun 15 07:34:46.841372 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:46.841336 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:47.018270 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:47.018217 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:48.075457 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:48.075431 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:48.253303 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:48.253271 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:49.389882 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:49.389854 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:49.574075 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:49.574043 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:50.604665 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:50.604638 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:50.775187 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:50.775155 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:51.805851 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:51.805823 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:51.993350 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:51.993307 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:53.041374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:53.041344 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:53.214572 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:53.214523 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:54.315960 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:54.315923 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:54.491552 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:54.491496 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:55.560321 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:55.560288 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:55.743092 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:55.743058 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:56.876635 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:56.876584 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:57.052970 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:57.052923 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:58.135614 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:58.135566 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:58.301602 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:58.301566 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:34:58.896527 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:58.896491 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:34:58.964963 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:58.964925 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:34:59.407751 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:59.407692 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:34:59.592022 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:34:59.591980 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:00.612337 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:00.612297 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:00.789171 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:00.789135 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:01.826648 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:01.826618 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:02.000589 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:02.000535 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:03.020624 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:03.020587 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:03.201274 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:03.201239 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:04.313025 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:04.312981 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:04.483044 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:04.483009 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:05.577649 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:05.577612 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:05.756601 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:05.756564 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:06.770017 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:06.769959 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:06.940045 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:06.940008 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:07.964874 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:07.964847 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:08.151159 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:08.151122 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:09.216782 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:09.216746 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:09.382901 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:09.382863 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:10.432291 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:10.432257 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:10.602552 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:10.602520 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:11.621160 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:11.621118 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:11.807575 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:11.807515 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:12.822058 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:12.822027 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:12.992958 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:12.992914 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:14.073076 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:14.073043 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:14.280708 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:14.280674 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:15.345912 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:15.345884 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:15.525674 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:15.525641 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:16.564898 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:16.564869 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:16.744486 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:16.744447 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:17.785078 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:17.785044 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:17.978007 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:17.977975 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:19.008031 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:19.008003 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:19.189529 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:19.189483 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:20.241450 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:20.241417 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:20.421303 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:20.421275 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:21.492124 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:21.492089 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:21.670434 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:21.670401 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:22.698914 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:22.698886 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:22.871814 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:22.871784 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:23.919430 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:23.919402 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:24.112321 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:24.112287 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:25.149409 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:25.149370 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:25.321527 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:25.321494 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:26.359659 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:26.359629 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:26.538031 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:26.537991 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:27.572871 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:27.572843 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:27.755766 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:27.755720 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:28.789984 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:28.789954 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:28.959438 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:28.959408 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:30.059730 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:30.059033 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:30.230873 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:30.230837 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:31.320741 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:31.320711 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:31.496919 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:31.496871 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:32.628430 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:32.628393 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:32.811096 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:32.811054 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:33.900261 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:33.900216 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:34.076174 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:34.076140 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:35.126971 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:35.126915 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:35.313198 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:35.313158 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:36.388173 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:36.388146 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:36.559119 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:36.559076 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:37.598770 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:37.598722 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:37.773610 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:37.773572 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:38.876117 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:38.876083 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:39.077664 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:39.077630 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:40.139768 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:40.139739 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:40.319243 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:40.319205 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:41.425466 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:41.425432 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:41.602553 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:41.602510 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:42.709603 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:42.709575 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:42.887529 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:42.887497 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:43.922062 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:43.922030 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:44.104748 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:44.104702 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:45.154345 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:45.154318 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:45.327518 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:45.327482 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:46.377689 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:46.377659 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:46.555954 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:46.555919 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:49.005557 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:49.005508 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:35:49.176408 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:49.176375 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:35:49.246674 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:49.246637 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:35:51.997684 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:51.997655 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pvj8k_9039cf21-b4dc-4e30-bdb6-a08444213d19/global-pull-secret-syncer/0.log" Jun 15 07:35:52.090335 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:52.090301 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" Jun 15 07:35:52.259591 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:52.259493 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-226.ec2.internal_f281e2e25ec3badbb2247e6dadbdfaf5/haproxy/0.log" Jun 15 07:35:56.049735 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:56.049685 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:35:56.149118 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:56.149081 2569 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4c5_338e9902-5e59-4baf-a031-c3b32ce89c50/konnectivity-agent/0.log" limit=1024 Jun 15 07:35:56.810484 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:56.810452 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-knbkx_6e5a3783-7faf-453d-92d5-cf282e5dc44e/manager/0.log" Jun 15 07:35:57.031693 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:57.031662 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5c9bd5678d-xwtf4_75fc04a5-2d46-4445-8d6a-6fbb4e3bc41b/manager/0.log" Jun 15 07:35:58.532905 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.532875 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9bea2f72-a04f-484d-97b5-6d3d2bee1c64/alertmanager/0.log" Jun 15 07:35:58.571797 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.571765 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9bea2f72-a04f-484d-97b5-6d3d2bee1c64/config-reloader/0.log" Jun 15 07:35:58.600409 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.600382 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9bea2f72-a04f-484d-97b5-6d3d2bee1c64/kube-rbac-proxy-web/0.log" Jun 15 07:35:58.634335 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.634287 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9bea2f72-a04f-484d-97b5-6d3d2bee1c64/kube-rbac-proxy/0.log" Jun 15 07:35:58.658616 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.658587 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9bea2f72-a04f-484d-97b5-6d3d2bee1c64/kube-rbac-proxy-metric/0.log" Jun 15 07:35:58.688256 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.688230 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9bea2f72-a04f-484d-97b5-6d3d2bee1c64/prom-label-proxy/0.log" Jun 15 07:35:58.723337 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.723307 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9bea2f72-a04f-484d-97b5-6d3d2bee1c64/init-config-reloader/0.log" Jun 15 07:35:58.825112 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.825025 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-9xm6l_d190abf6-db4f-42e4-a063-87d280b4e39e/kube-state-metrics/0.log" Jun 15 07:35:58.849453 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.849430 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-9xm6l_d190abf6-db4f-42e4-a063-87d280b4e39e/kube-rbac-proxy-main/0.log" Jun 15 07:35:58.879678 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.879652 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-9xm6l_d190abf6-db4f-42e4-a063-87d280b4e39e/kube-rbac-proxy-self/0.log" Jun 15 07:35:58.914340 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.914309 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-796477895d-ppfwg_9ab23e3e-6a4e-4a7a-bc5c-618bb344a263/metrics-server/0.log" Jun 15 07:35:58.989150 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:58.989123 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2k5fq_fd928336-049e-4cca-88a1-bfdeb087f7f9/node-exporter/0.log" Jun 15 07:35:59.027084 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.027059 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2k5fq_fd928336-049e-4cca-88a1-bfdeb087f7f9/kube-rbac-proxy/0.log" Jun 15 07:35:59.050151 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.050127 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2k5fq_fd928336-049e-4cca-88a1-bfdeb087f7f9/init-textfile/0.log" Jun 15 07:35:59.558371 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.558311 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81f4c287-57a0-4cce-bd05-db5ea473b276/prometheus/0.log" Jun 15 07:35:59.590598 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.590559 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81f4c287-57a0-4cce-bd05-db5ea473b276/config-reloader/0.log" Jun 15 07:35:59.635788 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.635755 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81f4c287-57a0-4cce-bd05-db5ea473b276/thanos-sidecar/0.log" Jun 15 07:35:59.729390 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.729365 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81f4c287-57a0-4cce-bd05-db5ea473b276/kube-rbac-proxy-web/0.log" Jun 15 07:35:59.780236 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.780210 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81f4c287-57a0-4cce-bd05-db5ea473b276/kube-rbac-proxy/0.log" Jun 15 07:35:59.873316 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.873207 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81f4c287-57a0-4cce-bd05-db5ea473b276/kube-rbac-proxy-thanos/0.log" Jun 15 07:35:59.925046 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.925022 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_81f4c287-57a0-4cce-bd05-db5ea473b276/init-config-reloader/0.log" Jun 15 07:35:59.967299 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:35:59.967271 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-864fcfc6f9-jdntm_144f3ad2-f47e-47aa-b482-7f4d0fbd473f/prometheus-operator/0.log" Jun 15 07:36:00.009808 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:00.009777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-864fcfc6f9-jdntm_144f3ad2-f47e-47aa-b482-7f4d0fbd473f/kube-rbac-proxy/0.log" Jun 15 07:36:00.048218 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:00.048187 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5cf667749c-fkznk_21666edb-c19b-470b-9ae8-75b6dee41a50/prometheus-operator-admission-webhook/0.log" Jun 15 07:36:02.438969 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.438931 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b"] Jun 15 07:36:02.439359 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439339 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="storage-initializer" Jun 15 07:36:02.439359 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439351 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="storage-initializer" Jun 15 07:36:02.439483 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439364 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="main" Jun 15 07:36:02.439483 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439370 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="main" Jun 15 07:36:02.439483 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439377 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="tokenizer" Jun 15 07:36:02.439483 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439382 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="tokenizer" Jun 15 07:36:02.439483 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439463 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="main" Jun 15 07:36:02.439483 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.439473 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="be319099-87c8-4296-b22c-cf490bb80894" containerName="tokenizer" Jun 15 07:36:02.442910 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.442889 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.445744 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.445725 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d7tsl\"/\"default-dockercfg-vgmh8\"" Jun 15 07:36:02.446867 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.446839 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d7tsl\"/\"openshift-service-ca.crt\"" Jun 15 07:36:02.446983 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.446852 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d7tsl\"/\"kube-root-ca.crt\"" Jun 15 07:36:02.462804 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.462774 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b"] Jun 15 07:36:02.514634 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.514597 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzl7z\" (UniqueName: \"kubernetes.io/projected/63b2a614-0eb6-4e4d-89ac-2b6333817fff-kube-api-access-dzl7z\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.514818 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.514640 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-sys\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.514818 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.514751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-podres\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.514818 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.514785 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-proc\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.514954 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.514829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-lib-modules\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616183 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-podres\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-proc\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-lib-modules\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616319 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-podres\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616374 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616371 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzl7z\" (UniqueName: \"kubernetes.io/projected/63b2a614-0eb6-4e4d-89ac-2b6333817fff-kube-api-access-dzl7z\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616599 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616403 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-sys\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616599 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616419 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-lib-modules\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616599 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616466 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-sys\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.616599 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.616513 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/63b2a614-0eb6-4e4d-89ac-2b6333817fff-proc\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.626428 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.626397 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzl7z\" (UniqueName: \"kubernetes.io/projected/63b2a614-0eb6-4e4d-89ac-2b6333817fff-kube-api-access-dzl7z\") pod \"perf-node-gather-daemonset-jg94b\" (UID: \"63b2a614-0eb6-4e4d-89ac-2b6333817fff\") " pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.753679 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.753588 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:02.905004 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:02.904966 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b"] Jun 15 07:36:02.906455 ip-10-0-128-226 kubenswrapper[2569]: W0615 07:36:02.906423 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod63b2a614_0eb6_4e4d_89ac_2b6333817fff.slice/crio-11d00ed15a8c2dbbb2fcec8608b8beb76e4207348b1e4aaeea4bd9608f7b5a40 WatchSource:0}: Error finding container 11d00ed15a8c2dbbb2fcec8608b8beb76e4207348b1e4aaeea4bd9608f7b5a40: Status 404 returned error can't find the container with id 11d00ed15a8c2dbbb2fcec8608b8beb76e4207348b1e4aaeea4bd9608f7b5a40 Jun 15 07:36:03.490612 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:03.490577 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" event={"ID":"63b2a614-0eb6-4e4d-89ac-2b6333817fff","Type":"ContainerStarted","Data":"299acb3703de17bda578e1c0b81e25faeb6da9b05ee4a45a226464c43d78e2e7"} Jun 15 07:36:03.491000 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:03.490631 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" event={"ID":"63b2a614-0eb6-4e4d-89ac-2b6333817fff","Type":"ContainerStarted","Data":"11d00ed15a8c2dbbb2fcec8608b8beb76e4207348b1e4aaeea4bd9608f7b5a40"} Jun 15 07:36:03.491000 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:03.490663 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:03.514260 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:03.514196 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" podStartSLOduration=1.514176327 podStartE2EDuration="1.514176327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 07:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 07:36:03.512225305 +0000 UTC m=+6441.857428487" watchObservedRunningTime="2026-06-15 07:36:03.514176327 +0000 UTC m=+6441.859379469" Jun 15 07:36:05.202512 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:05.202485 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d97j4_d4c07138-376d-46c4-9ac5-85733a50439a/dns/0.log" Jun 15 07:36:05.247634 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:05.247609 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d97j4_d4c07138-376d-46c4-9ac5-85733a50439a/kube-rbac-proxy/0.log" Jun 15 07:36:05.543571 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:05.543512 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqzpb_a19a3731-5372-4d37-83aa-6474aad11f0e/dns-node-resolver/0.log" Jun 15 07:36:06.273314 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:06.273280 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g28pn_2be932e7-3cd4-48b1-88ac-e9cead0e0a1e/node-ca/0.log" Jun 15 07:36:09.173775 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:09.173744 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fmtjs_312ffed8-5109-4ae0-9b62-74f951fc8068/serve-healthcheck-canary/0.log" Jun 15 07:36:09.506902 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:09.506815 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d7tsl/perf-node-gather-daemonset-jg94b" Jun 15 07:36:09.787717 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:09.787687 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bx7hl_914454af-bf9e-49b0-bfbb-e0d5d551d834/kube-rbac-proxy/0.log" Jun 15 07:36:09.854178 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:09.854142 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bx7hl_914454af-bf9e-49b0-bfbb-e0d5d551d834/exporter/0.log" Jun 15 07:36:09.897487 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:09.897459 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bx7hl_914454af-bf9e-49b0-bfbb-e0d5d551d834/extractor/0.log" Jun 15 07:36:13.362317 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:13.362291 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74f47976d9-v6qw2_bb11ea56-a48d-4258-ab94-d664c9ceb3f8/manager/0.log" Jun 15 07:36:14.177405 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:14.177376 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-x2wgq_ba617900-b001-4d38-8840-74301c5f4a0d/server/0.log" Jun 15 07:36:14.645040 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:14.645005 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-z4qqr_9d4ba447-7c6d-443f-9ac1-e057420d6bfa/manager/0.log" Jun 15 07:36:14.679567 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:14.679514 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rpbr4_58933d47-3efe-4d9d-96be-e807e88f213e/s3-init/0.log" Jun 15 07:36:14.699252 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:14.699228 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rpbr4_58933d47-3efe-4d9d-96be-e807e88f213e/download-hf-model/0.log" Jun 15 07:36:20.036455 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:20.036429 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-8dt2x_eef77ce7-de11-4e1e-83c3-6393e0628087/migrator/0.log" Jun 15 07:36:20.063453 ip-10-0-128-226 kubenswrapper[2569]: I0615 07:36:20.063411 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-8dt2x_eef77ce7-de11-4e1e-83c3-6393e0628087/graceful-termination/0.log"