Jul 09 17:37:18.502558 ip-10-0-141-44 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 09 17:37:18.502570 ip-10-0-141-44 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 09 17:37:18.502580 ip-10-0-141-44 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 09 17:37:18.502882 ip-10-0-141-44 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 09 17:37:28.584548 ip-10-0-141-44 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 09 17:37:28.584571 ip-10-0-141-44 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 8d31b2d0bb9c4666afd286ead62b3372 --
Jul 09 17:39:25.696083 ip-10-0-141-44 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.115173    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119257    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119267    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119271    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119274    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119277    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119281    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119284    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:26.211796 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119287    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:26.180592 ip-10-0-141-44 systemd[1]: Started Kubernetes Kubelet.
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119289    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119292    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119295    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119297    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119300    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119303    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119305    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119309    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119311    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119314    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119317    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119320    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119323    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119327    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119331    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119335    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119338    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119341    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:26.213259 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119344    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119347    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119349    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119352    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119354    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119357    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119360    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119363    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119366    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119368    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119372    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119375    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119378    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119380    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119383    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119385    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119388    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119391    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119393    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:26.214165 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119396    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119398    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119401    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119403    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119406    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119409    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119411    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119416    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119419    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119422    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119425    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119429    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119433    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119436    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119439    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119442    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119445    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119448    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119451    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:26.214989 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119454    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119456    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119459    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119461    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119464    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119467    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119469    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119472    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119475    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119478    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119480    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119483    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119486    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119489    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119491    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119494    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119496    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119499    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119502    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:26.215738 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119504    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119507    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119510    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119513    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119515    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119518    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119520    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119523    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119526    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119528    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119531    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119534    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119536    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119539    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119541    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.119544    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120240    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120247    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120251    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120254    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:26.216672 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120257    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120260    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120264    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120268    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120271    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120274    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120277    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120280    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120283    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120286    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120288    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120291    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120293    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120296    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120299    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120302    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120304    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120307    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120310    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:26.217740 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120312    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120315    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120318    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120320    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120323    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120328    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120331    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120333    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120336    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120339    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120341    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120344    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120346    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120349    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120351    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120354    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120356    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120359    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120361    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120372    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:26.218637 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120375    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120377    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120380    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120382    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120385    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120387    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120390    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120393    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120395    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120398    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120400    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120403    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120406    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120408    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120410    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120413    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120416    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120420    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120423    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120426    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:26.219642 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120429    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120432    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120435    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120437    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120439    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120442    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120445    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120447    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120450    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120453    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120455    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120460    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120464    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120467    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120470    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120473    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120476    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120478    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120481    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:26.220409 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120484    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120486    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120489    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120492    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120494    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120497    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120499    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120502    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120505    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120508    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120511    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120514    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120517    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120519    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120522    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120524    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120527    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120916    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120922    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120925    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:26.220987 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120927    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120932    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120936    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120939    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120942    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120945    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120948    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120951    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120954    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120956    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120960    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120964    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120967    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120969    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120972    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120977    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120979    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120983    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:26.221806 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120985    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120989    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120992    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120995    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.120999    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121002    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121005    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121008    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121010    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121013    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121015    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121018    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121021    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121023    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121026    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121028    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121031    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121033    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121036    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121038    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:26.222666 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121041    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121044    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121047    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121049    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121052    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121054    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121057    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121059    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121062    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121068    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121070    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121073    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121076    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121078    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121081    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121091    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121095    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121097    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121100    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:26.223508 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121103    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121106    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121108    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121111    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121113    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121117    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121119    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121122    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121124    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121126    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121129    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121132    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121134    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121136    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121140    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121142    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121145    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121147    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121150    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121152    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:26.224126 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121155    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121157    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121161    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121164    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121166    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121169    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121171    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121174    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121176    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121181    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121184    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121186    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121189    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121191    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121194    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121196    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121198    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121201    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121203    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121269    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121275    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 17:39:26.224751 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121280    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121286    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121291    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121296    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121302    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121307    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121314    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121318    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121321    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121324    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121327    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121330    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121333    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121337    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121340    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121343    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121347    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121350    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121354    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121356    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121361    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121365    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121368    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 17:39:26.225379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121371    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121374    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121377    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121379    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121382    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121385    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121388    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121392    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121395    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121398    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121401    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121404    2577 flags.go:64] FLAG: --enable-server="true"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121407    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121411    2577 flags.go:64] FLAG: --event-burst="100"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121414    2577 flags.go:64] FLAG: --event-qps="50"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121417    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121420    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121423    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121427    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121430    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121433    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121436    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121440    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121442    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121445    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 17:39:26.226030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121448    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121451    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121454    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121457    2577 flags.go:64] FLAG: --feature-gates=""
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121461    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121464    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121467    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121471    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121474    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121477    2577 flags.go:64] FLAG: --help="false"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121479    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121483    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121486    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121489    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121492    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121495    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121498    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121501    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121504    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121506    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121509    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121512    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121515    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121518    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 17:39:26.227138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121521    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121524    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121527    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121530    2577 flags.go:64] FLAG: --lock-file=""
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121532    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121536    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121539    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121544    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121547    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121549    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121552    2577 flags.go:64] FLAG: --logging-format="text"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121555    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121559    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121562    2577 flags.go:64] FLAG: --manifest-url=""
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121565    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121569    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121571    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121575    2577 flags.go:64] FLAG: --max-pods="110"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121578    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121581    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121584    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121587    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121590    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121593    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121596    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 17:39:26.230328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121603    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121606    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121609    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121612    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121615    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121620    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121623    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121626    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121629    2577 flags.go:64] FLAG: --port="10250"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121632    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121635    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f05f68d3b38e29b5"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121638    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121642    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121645    2577 flags.go:64] FLAG: --register-node="true"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121648    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121651    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121654    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121657    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121660    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121663    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121668    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121671    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121674    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121676    2577 flags.go:64] FLAG: --runonce="false"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121679    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 17:39:26.231200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121682    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121685    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121688    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121691    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121694    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121697    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121700    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121703    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121705    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121709    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121711    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121714    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121717    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121720    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121725    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121728    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121732    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121737    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121739    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121743    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121746    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121749    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121752    2577 flags.go:64] FLAG: --v="2"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121755    2577 flags.go:64] FLAG: --version="false"
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121760    2577 flags.go:64] FLAG: --vmodule=""
Jul 09 17:39:26.373618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121763    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.121766    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121893    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121899    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121903    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121906    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121909    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121912    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121916    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121920    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121923    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121926    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121929    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121932    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121934    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121937    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121940    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121942    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121945    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121948    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:26.399498 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121950    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121953    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121955    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121958    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121961    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121964    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121968    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121971    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121974    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121976    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121978    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121981    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121984    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121986    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121990    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121993    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121996    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.121998    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122001    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122003    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:26.400304 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122006    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122008    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122011    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122014    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122016    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122018    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122021    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122024    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122026    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122030    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122033    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122036    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122039    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122042    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122045    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122048    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122050    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122054    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122058    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:26.401114 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122061    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122064    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122067    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122070    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122072    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122075    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122078    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122081    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122084    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122087    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122089    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122092    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122094    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122097    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122100    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122102    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122105    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122108    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122110    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:26.401722 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122113    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122115    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122117    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122120    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122122    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122125    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122128    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122130    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122133    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122135    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122137    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122140    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122144    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122147    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122150    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122152    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122155    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122157    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122159    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122162    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:26.402331 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122165    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122168    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.122171    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.122788    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.129451    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.129468    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.130219    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.130228    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130302    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130310    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130314    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130319    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130323    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130328    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:26.402891 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130332    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130336    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130340    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130344    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130348    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130353    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130356    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130361    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130365    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130369    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130373    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130377    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130381    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130384    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130388    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130391    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130395    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130399    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130402    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130406    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:26.403351 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130412    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130417    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130421    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130426    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130430    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130434    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130438    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130442    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130445    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130450    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130454    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130458    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130462    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130466    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130470    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130473    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130477    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130480    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130485    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:26.403912 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130489    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130493    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130496    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130500    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130505    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130509    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130512    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130516    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130520    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130524    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130528    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130532    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130536    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130539    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130544    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130547    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130551    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130555    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130560    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130565    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:26.404464 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130572    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130575    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130580    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130583    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130587    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130591    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130595    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130598    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130602    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130606    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130609    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130613    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130617    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130620    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130626    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130631    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130636    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130640    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130645    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130649    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:26.405020 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130653    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130657    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130661    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130664    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130669    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130673    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130677    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130680    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130684    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130687    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130691    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130695    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130699    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130703    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.130710    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:26.405617 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130876    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130883    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130887    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130892    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130896    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130900    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130904    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130908    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130912    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130916    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130919    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130923    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130927    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130932    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130936    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130940    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130944    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130948    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130951    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130955    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 17:39:26.406061 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130959    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130962    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130966    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130970    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130973    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130976    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130980    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130984    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130987    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130992    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130996    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.130999    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131002    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131006    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131010    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131014    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131017    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131021    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131025    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 17:39:26.406664 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131031    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131036    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131039    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131044    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131048    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131051    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131057    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131061    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131064    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131069    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131073    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131076    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131080    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131084    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131088    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131091    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131095    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131099    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131102    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131107    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 17:39:26.407192 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131110    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131114    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131119    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131123    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131128    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131132    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131135    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131139    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131143    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131146    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131150    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131154    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131157    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131161    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131165    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131169    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131172    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131176    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131182    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131186    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 17:39:26.407777 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131190    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131193    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131198    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131202    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131206    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131210    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131213    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131217    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131220    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131225    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131228    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131232    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131236    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131240    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131248    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131253    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131257    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131263    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131268    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 17:39:26.408375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:26.131272    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.131278    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.131402    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.133718    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.134896    2577 server.go:1019] "Starting client certificate rotation"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.136064    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.136135    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.137382    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.139861    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.146740    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.150289    2577 log.go:25] "Validated CRI v1 image API"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.153347    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 17:39:26.409228 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.155559    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b133c146-897a-4af8-bfd5-724847e4731a:/dev/nvme0n1p4 d4f1761e-7254-4f4a-bf2b-0cd4e7635695:/dev/nvme0n1p3]
Jul 09 17:39:26.409770 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.155573    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 17:39:26.409997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.160811    2577 manager.go:216] Machine: {Timestamp:2026-07-09 17:39:26.159603583 +0000 UTC m=+0.349963489 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099825 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec298c9c0227b8f467a53407cc90a175 SystemUUID:ec298c9c-0227-b8f4-67a5-3407cc90a175 BootID:8d31b2d0-bb9c-4666-afd2-86ead62b3372 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:17:37:68:11:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:17:37:68:11:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:d8:a6:d9:7a:e6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 17:39:26.409997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.160929    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 17:39:26.409997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.160995    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 17:39:26.409997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.164267    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 17:39:26.409997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.164471    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:26.409997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.164290    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-44.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 17:39:26.409997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.165007    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.165017    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.165025    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.165546    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.166641    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170476    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170574    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170601    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170610    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170630    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170639    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170655    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.170985    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.171000    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.173438    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175123    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175150    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175289    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175307    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175317    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175324    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175333    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.175337    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176043    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176049    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176055    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176069    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176081    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176089    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176706    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.176715    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.179818    2577 server.go:1268] "Started kubelet"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.179891    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.180470    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.180653    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.181336    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-44.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.181364    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.181429    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.181638    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 17:39:26.411054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.182163    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.184364    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.186017    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-44.ec2.internal.18c0afd15cddf1b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-44.ec2.internal,UID:ip-10-0-141-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-44.ec2.internal,},FirstTimestamp:2026-07-09 17:39:26.179791283 +0000 UTC m=+0.370151190,LastTimestamp:2026-07-09 17:39:26.179791283 +0000 UTC m=+0.370151190,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-44.ec2.internal,}"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.186894    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.186903    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.187730    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.187891    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.188299    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.188311    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.188379    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.188617    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.189325    2577 factory.go:55] Registering systemd factory
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.189339    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.189594    2577 factory.go:153] Registering CRI-O factory
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.189605    2577 factory.go:223] Registration of the crio container factory successfully
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.195080    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.195113    2577 factory.go:103] Registering Raw factory
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.199093    2577 manager.go:1196] Started watching for new ooms in manager
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.200337    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-44.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.200341    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.200446    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tnvrw"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.200510    2577 manager.go:349] Starting recovery of all containers
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.205401    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.207599    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tnvrw"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.220568    2577 manager.go:354] Recovery completed
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.224728    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.226676    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.226695    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.226704    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.227242    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.227249    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.227268    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.229286    2577 policy_none.go:49] "None policy: Start"
Jul 09 17:39:26.566122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.229297    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.229306    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.231284    2577 policy_none.go:47] "Start"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.231855    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.232149    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-44.ec2.internal.18c0afd15fa982cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-44.ec2.internal,UID:ip-10-0-141-44.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-44.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-44.ec2.internal,},FirstTimestamp:2026-07-09 17:39:26.226686667 +0000 UTC m=+0.417046575,LastTimestamp:2026-07-09 17:39:26.226686667 +0000 UTC m=+0.417046575,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-44.ec2.internal,}"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.232688    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.232714    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.232730    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.268391    2577 manager.go:337] "Starting Device Plugin manager"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.268520    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.268532    2577 server.go:87] "Starting device plugin registration server"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.268708    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.268716    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.268796    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.268896    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.268905    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.269368    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.269413    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.368805    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.369540    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.369564    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.369574    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.369591    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.387249    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.387265    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-44.ec2.internal\": node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.411468    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.411514    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.411538    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.411577    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.414739    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.420964    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.513000    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"]
Jul 09 17:39:26.568617 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.513060    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.513921    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.513951    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.513966    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.516158    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.516308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.516339    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.517684    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.517698    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.517717    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.517704    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.517751    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.517727    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.519963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.519989    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.520793    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.520815    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.520827    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeHasSufficientPID"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.521003    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.542437    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-44.ec2.internal\" not found" node="ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.569662 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.550945    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-44.ec2.internal\" not found" node="ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.592290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89b829e494294382f2ba3cb0e5e2f900-config\") pod \"kube-apiserver-proxy-ip-10-0-141-44.ec2.internal\" (UID: \"89b829e494294382f2ba3cb0e5e2f900\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.592310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09aaeda5a43a1bc9c803c4ce38e0e34d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"09aaeda5a43a1bc9c803c4ce38e0e34d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.592326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09aaeda5a43a1bc9c803c4ce38e0e34d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"09aaeda5a43a1bc9c803c4ce38e0e34d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.621960    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.693232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09aaeda5a43a1bc9c803c4ce38e0e34d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"09aaeda5a43a1bc9c803c4ce38e0e34d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.693256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09aaeda5a43a1bc9c803c4ce38e0e34d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"09aaeda5a43a1bc9c803c4ce38e0e34d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.693273    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89b829e494294382f2ba3cb0e5e2f900-config\") pod \"kube-apiserver-proxy-ip-10-0-141-44.ec2.internal\" (UID: \"89b829e494294382f2ba3cb0e5e2f900\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.693299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/89b829e494294382f2ba3cb0e5e2f900-config\") pod \"kube-apiserver-proxy-ip-10-0-141-44.ec2.internal\" (UID: \"89b829e494294382f2ba3cb0e5e2f900\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.693328    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09aaeda5a43a1bc9c803c4ce38e0e34d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"09aaeda5a43a1bc9c803c4ce38e0e34d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.693331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/09aaeda5a43a1bc9c803c4ce38e0e34d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal\" (UID: \"09aaeda5a43a1bc9c803c4ce38e0e34d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.737342 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.722812    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.910809 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.824641    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:26.910809 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.845315    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:26.910809 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:26.854089    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:27.085100 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:26.925695    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:27.085100 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:27.027740    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:27.085100 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.042015    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.108052    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:27.128252    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.136799    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.136912    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.136950    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.136963    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.217154    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:34:27 +0000 UTC" deadline="2027-12-14 07:43:01.447543232 +0000 UTC"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.217184    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12542h3m34.230362028s"
Jul 09 17:39:27.257331 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:27.229073    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:27.393898 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:27.329431    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:27.516648 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:27.429895    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-44.ec2.internal\" not found"
Jul 09 17:39:27.516648 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.466391    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:27.516648 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.489135    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:27.520788 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.520495    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 17:39:27.688291 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.521713    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal"
Jul 09 17:39:27.688291 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.537623    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 17:39:27.688291 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:27.645164    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09aaeda5a43a1bc9c803c4ce38e0e34d.slice/crio-ce8e98b19ba8c211a8f6e0680c4cde3ad81ecf3f09774dc6d086aab24e521406 WatchSource:0}: Error finding container ce8e98b19ba8c211a8f6e0680c4cde3ad81ecf3f09774dc6d086aab24e521406: Status 404 returned error can't find the container with id ce8e98b19ba8c211a8f6e0680c4cde3ad81ecf3f09774dc6d086aab24e521406
Jul 09 17:39:27.688291 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:27.645715    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b829e494294382f2ba3cb0e5e2f900.slice/crio-b5cc8d9fe69f2284e9b802191f2f15fdce34cd2595f19821932026ee26b78b27 WatchSource:0}: Error finding container b5cc8d9fe69f2284e9b802191f2f15fdce34cd2595f19821932026ee26b78b27: Status 404 returned error can't find the container with id b5cc8d9fe69f2284e9b802191f2f15fdce34cd2595f19821932026ee26b78b27
Jul 09 17:39:27.688291 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:27.650294    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:39:28.194705 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.117361    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:28.194705 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.183099    2577 apiserver.go:52] "Watching apiserver"
Jul 09 17:39:28.194705 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.187397    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.206671    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.208982    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ns9fg","openshift-multus/network-metrics-daemon-gwxtd","openshift-network-diagnostics/network-check-target-dqvff","openshift-network-operator/iptables-alerter-tdqd5","openshift-ovn-kubernetes/ovnkube-node-sh5ck","kube-system/konnectivity-agent-lcjp2","kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm","openshift-cluster-node-tuning-operator/tuned-fp6j9","openshift-dns/node-resolver-8j9n9","openshift-image-registry/node-ca-m9ntl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal","openshift-multus/multus-additional-cni-plugins-t4m8f","kube-system/global-pull-secret-syncer-s45x2"]
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.211670    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.214011    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.216458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.216537    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.218911    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.222294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.227819    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.227973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.230042    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.230171    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.232662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.232728    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.235393    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.237450    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239315    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239366    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d2bzg\""
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239399    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239562    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239566    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239641    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 17:39:28.267196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239724    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239803    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.239942    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240013    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z2k6w\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240638    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240645    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f4j8k\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240740    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qn497\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240752    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240755    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240880    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.240934    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241024    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241077    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241172    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241198    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241336    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s7zhl\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241464    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241598    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4cmgp\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241704    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-q565w\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.241780    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.242150    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.255574    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.255612    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.256003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cfpk7\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.256130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z55rt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.256471    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.256585    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 17:39:28.270748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.256749    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 17:39:28.291418 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.274472    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xxzkm"
Jul 09 17:39:28.291418 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.274491    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xxzkm"
Jul 09 17:39:28.291418 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.289996    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c30946-998e-4d0d-8673-a898a8247bd7-tmp-dir\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304538    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-run-netns\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-cni-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-run\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304689    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-cni-netd\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de5fad1c-2d42-4610-97f3-595c50f0158e-konnectivity-ca\") pod \"konnectivity-agent-lcjp2\" (UID: \"de5fad1c-2d42-4610-97f3-595c50f0158e\") " pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-lib-modules\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-os-release\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-daemon-config\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304825    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkp4z\" (UniqueName: \"kubernetes.io/projected/38d9e254-7406-42bf-ab9c-97ede1271222-kube-api-access-qkp4z\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.315222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-hostroot\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-systemd-units\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-var-lib-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.304988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-os-release\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-tmp\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305035    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-cni-multus\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-kubelet\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8q9x\" (UniqueName: \"kubernetes.io/projected/25c30946-998e-4d0d-8673-a898a8247bd7-kube-api-access-q8q9x\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-ovn\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305151    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-host\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-conf-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-node-log\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-cni-bin\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.316593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysctl-d\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-cnibin\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305392    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-log-socket\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305452    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-cnibin\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjm9\" (UniqueName: \"kubernetes.io/projected/7e562981-a609-4972-95ee-ca9b7095b4f5-kube-api-access-mtjm9\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-sys\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysconfig\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25c30946-998e-4d0d-8673-a898a8247bd7-hosts-file\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-system-cni-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-cni-binary-copy\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-socket-dir-parent\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.319709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305779    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-cni-bin\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-var-lib-kubelet\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305844    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5x2\" (UniqueName: \"kubernetes.io/projected/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-kube-api-access-xd5x2\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovnkube-config\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-sys-fs\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltcr4\" (UniqueName: \"kubernetes.io/projected/35527b65-4b17-4865-a944-d581cd153815-kube-api-access-ltcr4\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.305982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovnkube-script-lib\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gslp\" (UniqueName: \"kubernetes.io/projected/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-kube-api-access-4gslp\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-kubelet-config\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306049    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-dbus\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d9e254-7406-42bf-ab9c-97ede1271222-host-slash\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-modprobe-d\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.321495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-etc-kubernetes\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpptc\" (UniqueName: \"kubernetes.io/projected/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-kube-api-access-dpptc\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-socket-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306244    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-registration-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-device-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-etc-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306364    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-k8s-cni-cncf-io\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306389    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-kubelet\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-env-overrides\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de5fad1c-2d42-4610-97f3-595c50f0158e-agent-certs\") pod \"konnectivity-agent-lcjp2\" (UID: \"de5fad1c-2d42-4610-97f3-595c50f0158e\") " pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52mxs\" (UniqueName: \"kubernetes.io/projected/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-kube-api-access-52mxs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:28.323366 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306567    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-netns\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-kubernetes\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-systemd\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/338f5cc2-493a-43df-8aa6-a1e4b609de1f-serviceca\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-slash\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-tuned\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306752    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/338f5cc2-493a-43df-8aa6-a1e4b609de1f-host\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbnd\" (UniqueName: \"kubernetes.io/projected/338f5cc2-493a-43df-8aa6-a1e4b609de1f-kube-api-access-xbbnd\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-systemd\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovn-node-metrics-cert\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.306978    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38d9e254-7406-42bf-ab9c-97ede1271222-iptables-alerter-script\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.324279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.307002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysctl-conf\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.325635 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.307028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-multus-certs\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407214    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c30946-998e-4d0d-8673-a898a8247bd7-tmp-dir\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-run-netns\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-cni-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407331    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-run\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407354    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-cni-netd\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de5fad1c-2d42-4610-97f3-595c50f0158e-konnectivity-ca\") pod \"konnectivity-agent-lcjp2\" (UID: \"de5fad1c-2d42-4610-97f3-595c50f0158e\") " pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-lib-modules\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-os-release\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-daemon-config\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407525    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkp4z\" (UniqueName: \"kubernetes.io/projected/38d9e254-7406-42bf-ab9c-97ede1271222-kube-api-access-qkp4z\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-hostroot\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.476293 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-systemd-units\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407603    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-var-lib-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-os-release\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-tmp\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-cni-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407675    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-cni-multus\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-kubelet\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8q9x\" (UniqueName: \"kubernetes.io/projected/25c30946-998e-4d0d-8673-a898a8247bd7-kube-api-access-q8q9x\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-ovn\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-host\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-conf-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-node-log\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-cni-bin\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.407893    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysctl-d\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.480272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-cnibin\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-run-netns\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-var-lib-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-log-socket\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-cni-netd\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-ovn\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408894    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-cni-multus\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408896    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/de5fad1c-2d42-4610-97f3-595c50f0158e-konnectivity-ca\") pod \"konnectivity-agent-lcjp2\" (UID: \"de5fad1c-2d42-4610-97f3-595c50f0158e\") " pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-cnibin\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-kubelet\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408985    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 17:39:28.481241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjm9\" (UniqueName: \"kubernetes.io/projected/7e562981-a609-4972-95ee-ca9b7095b4f5-kube-api-access-mtjm9\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysctl-d\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-node-log\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-hostroot\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-cni-bin\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409128    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-systemd-units\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-daemon-config\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-sys\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-run-ovn-kubernetes\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408005    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c30946-998e-4d0d-8673-a898a8247bd7-tmp-dir\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-sys\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysconfig\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25c30946-998e-4d0d-8673-a898a8247bd7-hosts-file\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409717    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.482003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408275    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-run\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409787    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysconfig\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25c30946-998e-4d0d-8673-a898a8247bd7-hosts-file\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-os-release\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-lib-modules\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.408974    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-os-release\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409048    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-host\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-cnibin\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-system-cni-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-log-socket\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7e562981-a609-4972-95ee-ca9b7095b4f5-cnibin\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-cni-binary-copy\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-system-cni-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.484232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410066    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-socket-dir-parent\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.409011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-conf-dir\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-cni-bin\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-var-lib-kubelet\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-multus-socket-dir-parent\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5x2\" (UniqueName: \"kubernetes.io/projected/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-kube-api-access-xd5x2\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-var-lib-cni-bin\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovnkube-config\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-sys-fs\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-var-lib-kubelet\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltcr4\" (UniqueName: \"kubernetes.io/projected/35527b65-4b17-4865-a944-d581cd153815-kube-api-access-ltcr4\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410299    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovnkube-script-lib\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gslp\" (UniqueName: \"kubernetes.io/projected/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-kube-api-access-4gslp\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.485176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410379    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-kubelet-config\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-sys-fs\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-dbus\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d9e254-7406-42bf-ab9c-97ede1271222-host-slash\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-modprobe-d\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-etc-kubernetes\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpptc\" (UniqueName: \"kubernetes.io/projected/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-kube-api-access-dpptc\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-cni-binary-copy\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-socket-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-registration-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410601    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-dbus\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-device-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-device-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.486566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410763    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-modprobe-d\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovnkube-config\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410825    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-registration-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410836    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-kubelet-config\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-etc-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-k8s-cni-cncf-io\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410934    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-etc-kubernetes\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.410983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/38d9e254-7406-42bf-ab9c-97ede1271222-host-slash\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411024    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-etc-openvswitch\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-etc-selinux\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/35527b65-4b17-4865-a944-d581cd153815-socket-dir\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-k8s-cni-cncf-io\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-kubelet\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.487525 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-kubelet\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411128    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-env-overrides\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de5fad1c-2d42-4610-97f3-595c50f0158e-agent-certs\") pod \"konnectivity-agent-lcjp2\" (UID: \"de5fad1c-2d42-4610-97f3-595c50f0158e\") " pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411159    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovnkube-script-lib\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411211    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52mxs\" (UniqueName: \"kubernetes.io/projected/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-kube-api-access-52mxs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411264    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-netns\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-kubernetes\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411313    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-systemd\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.411329    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411338    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/338f5cc2-493a-43df-8aa6-a1e4b609de1f-serviceca\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-slash\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-netns\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.488445 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.411410    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs podName:2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:28.911373287 +0000 UTC m=+3.101733184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs") pod "network-metrics-daemon-gwxtd" (UID: "2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-slash\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411431    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-tuned\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-kubernetes\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-systemd\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/338f5cc2-493a-43df-8aa6-a1e4b609de1f-host\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbnd\" (UniqueName: \"kubernetes.io/projected/338f5cc2-493a-43df-8aa6-a1e4b609de1f-kube-api-access-xbbnd\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411729    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-systemd\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411782    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovn-node-metrics-cert\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38d9e254-7406-42bf-ab9c-97ede1271222-iptables-alerter-script\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysctl-conf\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.489863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411919    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-multus-certs\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411978    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-env-overrides\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.411980    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-host-run-multus-certs\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.412027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-run-systemd\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.412888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/338f5cc2-493a-43df-8aa6-a1e4b609de1f-serviceca\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.413050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-sysctl-conf\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.413094    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/338f5cc2-493a-43df-8aa6-a1e4b609de1f-host\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.413162    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.413233    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret podName:9af1998d-6f11-4d4c-b45c-bcf73f88e1a1 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:28.913214677 +0000 UTC m=+3.103574586 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret") pod "global-pull-secret-syncer-s45x2" (UID: "9af1998d-6f11-4d4c-b45c-bcf73f88e1a1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.413629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/38d9e254-7406-42bf-ab9c-97ede1271222-iptables-alerter-script\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.413898    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7e562981-a609-4972-95ee-ca9b7095b4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.415667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-tmp\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.415776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-ovn-node-metrics-cert\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.417157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-etc-tuned\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.417783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/de5fad1c-2d42-4610-97f3-595c50f0158e-agent-certs\") pod \"konnectivity-agent-lcjp2\" (UID: \"de5fad1c-2d42-4610-97f3-595c50f0158e\") " pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.490517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.418419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" event={"ID":"09aaeda5a43a1bc9c803c4ce38e0e34d","Type":"ContainerStarted","Data":"ce8e98b19ba8c211a8f6e0680c4cde3ad81ecf3f09774dc6d086aab24e521406"}
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.420432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal" event={"ID":"89b829e494294382f2ba3cb0e5e2f900","Type":"ContainerStarted","Data":"b5cc8d9fe69f2284e9b802191f2f15fdce34cd2595f19821932026ee26b78b27"}
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.434970    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.434995    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.435009    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7z4bg for pod openshift-network-diagnostics/network-check-target-dqvff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.435060    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg podName:582ab465-95af-4733-970f-8e3cee8ca49e nodeName:}" failed. No retries permitted until 2026-07-09 17:39:28.935044732 +0000 UTC m=+3.125404644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7z4bg" (UniqueName: "kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg") pod "network-check-target-dqvff" (UID: "582ab465-95af-4733-970f-8e3cee8ca49e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.438948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gslp\" (UniqueName: \"kubernetes.io/projected/8abf9808-a4c1-4b03-8952-af5a3f1b63a3-kube-api-access-4gslp\") pod \"ovnkube-node-sh5ck\" (UID: \"8abf9808-a4c1-4b03-8952-af5a3f1b63a3\") " pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.440141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpptc\" (UniqueName: \"kubernetes.io/projected/67f6d6ab-4dfc-432a-85e7-7838ac2dba43-kube-api-access-dpptc\") pod \"multus-ns9fg\" (UID: \"67f6d6ab-4dfc-432a-85e7-7838ac2dba43\") " pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.441430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjm9\" (UniqueName: \"kubernetes.io/projected/7e562981-a609-4972-95ee-ca9b7095b4f5-kube-api-access-mtjm9\") pod \"multus-additional-cni-plugins-t4m8f\" (UID: \"7e562981-a609-4972-95ee-ca9b7095b4f5\") " pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.441841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkp4z\" (UniqueName: \"kubernetes.io/projected/38d9e254-7406-42bf-ab9c-97ede1271222-kube-api-access-qkp4z\") pod \"iptables-alerter-tdqd5\" (UID: \"38d9e254-7406-42bf-ab9c-97ede1271222\") " pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.442385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5x2\" (UniqueName: \"kubernetes.io/projected/5e7c5c76-19c3-41c8-a4f8-56d455d51dfa-kube-api-access-xd5x2\") pod \"tuned-fp6j9\" (UID: \"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa\") " pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.443279    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8q9x\" (UniqueName: \"kubernetes.io/projected/25c30946-998e-4d0d-8673-a898a8247bd7-kube-api-access-q8q9x\") pod \"node-resolver-8j9n9\" (UID: \"25c30946-998e-4d0d-8673-a898a8247bd7\") " pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.444356    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbnd\" (UniqueName: \"kubernetes.io/projected/338f5cc2-493a-43df-8aa6-a1e4b609de1f-kube-api-access-xbbnd\") pod \"node-ca-m9ntl\" (UID: \"338f5cc2-493a-43df-8aa6-a1e4b609de1f\") " pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.444828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52mxs\" (UniqueName: \"kubernetes.io/projected/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-kube-api-access-52mxs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:28.491175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.446468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltcr4\" (UniqueName: \"kubernetes.io/projected/35527b65-4b17-4865-a944-d581cd153815-kube-api-access-ltcr4\") pod \"aws-ebs-csi-driver-node-v2rjm\" (UID: \"35527b65-4b17-4865-a944-d581cd153815\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.541016 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.540990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8j9n9"
Jul 09 17:39:28.552167 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.552146    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t4m8f"
Jul 09 17:39:28.558124 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.558098    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25c30946_998e_4d0d_8673_a898a8247bd7.slice/crio-2e127d9708ae95455fed1ce7d40896d31c9c463b9465826c7d88e275694a0153 WatchSource:0}: Error finding container 2e127d9708ae95455fed1ce7d40896d31c9c463b9465826c7d88e275694a0153: Status 404 returned error can't find the container with id 2e127d9708ae95455fed1ce7d40896d31c9c463b9465826c7d88e275694a0153
Jul 09 17:39:28.563816 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.563791    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e562981_a609_4972_95ee_ca9b7095b4f5.slice/crio-c6077b50b85edc68634027a306701d72895d706bce4139c78250cb60b7aad7e6 WatchSource:0}: Error finding container c6077b50b85edc68634027a306701d72895d706bce4139c78250cb60b7aad7e6: Status 404 returned error can't find the container with id c6077b50b85edc68634027a306701d72895d706bce4139c78250cb60b7aad7e6
Jul 09 17:39:28.578554 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.578531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tdqd5"
Jul 09 17:39:28.584120 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.584096    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:28.587626 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.587604    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d9e254_7406_42bf_ab9c_97ede1271222.slice/crio-07985535f0bb0631066feb2d8f70a84c06069547a154220f4ffa5bdb7fe49827 WatchSource:0}: Error finding container 07985535f0bb0631066feb2d8f70a84c06069547a154220f4ffa5bdb7fe49827: Status 404 returned error can't find the container with id 07985535f0bb0631066feb2d8f70a84c06069547a154220f4ffa5bdb7fe49827
Jul 09 17:39:28.594734 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.594714    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:28.595001 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.594973    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abf9808_a4c1_4b03_8952_af5a3f1b63a3.slice/crio-4d5efbd6c379262b913c7c06d2c53fffef2b5dd05f04bd23fed9405e59009a0f WatchSource:0}: Error finding container 4d5efbd6c379262b913c7c06d2c53fffef2b5dd05f04bd23fed9405e59009a0f: Status 404 returned error can't find the container with id 4d5efbd6c379262b913c7c06d2c53fffef2b5dd05f04bd23fed9405e59009a0f
Jul 09 17:39:28.602039 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.602007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fp6j9"
Jul 09 17:39:28.607080 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.607056    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5fad1c_2d42_4610_97f3_595c50f0158e.slice/crio-d098d4c6115931c0ac2e866c5d191e01727b00c88f12870779545201351b7678 WatchSource:0}: Error finding container d098d4c6115931c0ac2e866c5d191e01727b00c88f12870779545201351b7678: Status 404 returned error can't find the container with id d098d4c6115931c0ac2e866c5d191e01727b00c88f12870779545201351b7678
Jul 09 17:39:28.613077 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.613038    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7c5c76_19c3_41c8_a4f8_56d455d51dfa.slice/crio-03a1582459d485b7c8b335af25db5b4cb5109cb0fa22c23b6664631fb7482b23 WatchSource:0}: Error finding container 03a1582459d485b7c8b335af25db5b4cb5109cb0fa22c23b6664631fb7482b23: Status 404 returned error can't find the container with id 03a1582459d485b7c8b335af25db5b4cb5109cb0fa22c23b6664631fb7482b23
Jul 09 17:39:28.614349 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.614139    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ns9fg"
Jul 09 17:39:28.624445 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.624407    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m9ntl"
Jul 09 17:39:28.625358 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.625338    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f6d6ab_4dfc_432a_85e7_7838ac2dba43.slice/crio-b4e5fd0611f8d96d48cc704e518b614df2b58e14de8dc06935b12ed023f369fd WatchSource:0}: Error finding container b4e5fd0611f8d96d48cc704e518b614df2b58e14de8dc06935b12ed023f369fd: Status 404 returned error can't find the container with id b4e5fd0611f8d96d48cc704e518b614df2b58e14de8dc06935b12ed023f369fd
Jul 09 17:39:28.639357 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:28.639286    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338f5cc2_493a_43df_8aa6_a1e4b609de1f.slice/crio-9e1be1cb84e95a605e294c1aa81c37814f941d433216f3fe9e9ebb42ab96b170 WatchSource:0}: Error finding container 9e1be1cb84e95a605e294c1aa81c37814f941d433216f3fe9e9ebb42ab96b170: Status 404 returned error can't find the container with id 9e1be1cb84e95a605e294c1aa81c37814f941d433216f3fe9e9ebb42ab96b170
Jul 09 17:39:28.684357 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.684319    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm"
Jul 09 17:39:28.783385 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.783359    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:28.916215 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.916144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:28.916215 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:28.916180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:28.916372 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.916293    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:28.916372 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.916326    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:28.916372 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.916370    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret podName:9af1998d-6f11-4d4c-b45c-bcf73f88e1a1 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:29.91633702 +0000 UTC m=+4.106696914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret") pod "global-pull-secret-syncer-s45x2" (UID: "9af1998d-6f11-4d4c-b45c-bcf73f88e1a1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:28.916513 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:28.916392    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs podName:2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:29.916380419 +0000 UTC m=+4.106740316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs") pod "network-metrics-daemon-gwxtd" (UID: "2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:29.017185 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.017151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:29.017343 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.017311    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:29.017343 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.017330    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:29.017343 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.017342    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7z4bg for pod openshift-network-diagnostics/network-check-target-dqvff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:29.017462 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.017403    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg podName:582ab465-95af-4733-970f-8e3cee8ca49e nodeName:}" failed. No retries permitted until 2026-07-09 17:39:30.017384134 +0000 UTC m=+4.207744029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z4bg" (UniqueName: "kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg") pod "network-check-target-dqvff" (UID: "582ab465-95af-4733-970f-8e3cee8ca49e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:29.277741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.277615    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:29 +0000 UTC" deadline="2028-04-16 18:35:17.76397906 +0000 UTC"
Jul 09 17:39:29.277741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.277688    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15528h55m48.486295524s"
Jul 09 17:39:29.425200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.425160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerStarted","Data":"c6077b50b85edc68634027a306701d72895d706bce4139c78250cb60b7aad7e6"}
Jul 09 17:39:29.427217 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.427180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8j9n9" event={"ID":"25c30946-998e-4d0d-8673-a898a8247bd7","Type":"ContainerStarted","Data":"2e127d9708ae95455fed1ce7d40896d31c9c463b9465826c7d88e275694a0153"}
Jul 09 17:39:29.430338 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.430285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m9ntl" event={"ID":"338f5cc2-493a-43df-8aa6-a1e4b609de1f","Type":"ContainerStarted","Data":"9e1be1cb84e95a605e294c1aa81c37814f941d433216f3fe9e9ebb42ab96b170"}
Jul 09 17:39:29.433464 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.433427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ns9fg" event={"ID":"67f6d6ab-4dfc-432a-85e7-7838ac2dba43","Type":"ContainerStarted","Data":"b4e5fd0611f8d96d48cc704e518b614df2b58e14de8dc06935b12ed023f369fd"}
Jul 09 17:39:29.435985 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.435911    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fp6j9" event={"ID":"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa","Type":"ContainerStarted","Data":"03a1582459d485b7c8b335af25db5b4cb5109cb0fa22c23b6664631fb7482b23"}
Jul 09 17:39:29.437456 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.437383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lcjp2" event={"ID":"de5fad1c-2d42-4610-97f3-595c50f0158e","Type":"ContainerStarted","Data":"d098d4c6115931c0ac2e866c5d191e01727b00c88f12870779545201351b7678"}
Jul 09 17:39:29.439221 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.439112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"4d5efbd6c379262b913c7c06d2c53fffef2b5dd05f04bd23fed9405e59009a0f"}
Jul 09 17:39:29.442300 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.442218    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tdqd5" event={"ID":"38d9e254-7406-42bf-ab9c-97ede1271222","Type":"ContainerStarted","Data":"07985535f0bb0631066feb2d8f70a84c06069547a154220f4ffa5bdb7fe49827"}
Jul 09 17:39:29.449470 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.449418    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 17:39:29.924094 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.924056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:29.924320 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:29.924119    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:29.924320 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.924248    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:29.924320 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.924292    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:29.924555 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.924323    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs podName:2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:31.924298565 +0000 UTC m=+6.114658465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs") pod "network-metrics-daemon-gwxtd" (UID: "2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:29.924555 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:29.924345    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret podName:9af1998d-6f11-4d4c-b45c-bcf73f88e1a1 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:31.924334027 +0000 UTC m=+6.114693928 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret") pod "global-pull-secret-syncer-s45x2" (UID: "9af1998d-6f11-4d4c-b45c-bcf73f88e1a1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:30.025417 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:30.025361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:30.025613 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:30.025582    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:30.025613 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:30.025611    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:30.025772 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:30.025625    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7z4bg for pod openshift-network-diagnostics/network-check-target-dqvff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:30.025772 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:30.025690    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg podName:582ab465-95af-4733-970f-8e3cee8ca49e nodeName:}" failed. No retries permitted until 2026-07-09 17:39:32.025671215 +0000 UTC m=+6.216031124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z4bg" (UniqueName: "kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg") pod "network-check-target-dqvff" (UID: "582ab465-95af-4733-970f-8e3cee8ca49e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:30.210524 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:39:30.210319    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35527b65_4b17_4865_a944_d581cd153815.slice/crio-17e590c06d769b5481b91527fa859435c90262972c10bcddab57164ed1029e4e WatchSource:0}: Error finding container 17e590c06d769b5481b91527fa859435c90262972c10bcddab57164ed1029e4e: Status 404 returned error can't find the container with id 17e590c06d769b5481b91527fa859435c90262972c10bcddab57164ed1029e4e
Jul 09 17:39:30.278223 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:30.278182    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:29 +0000 UTC" deadline="2028-01-11 02:12:48.955662255 +0000 UTC"
Jul 09 17:39:30.278223 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:30.278223    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13208h33m18.677444045s"
Jul 09 17:39:30.413084 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:30.413054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:30.413247 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:30.413194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:30.413673 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:30.413646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:30.413782 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:30.413760    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:30.415134 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:30.414963    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:30.415134 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:30.415099    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:30.447683 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:30.447611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm" event={"ID":"35527b65-4b17-4865-a944-d581cd153815","Type":"ContainerStarted","Data":"17e590c06d769b5481b91527fa859435c90262972c10bcddab57164ed1029e4e"}
Jul 09 17:39:31.941543 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:31.940642    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:31.941543 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:31.940700    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:31.941543 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:31.940857    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:31.941543 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:31.940943    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret podName:9af1998d-6f11-4d4c-b45c-bcf73f88e1a1 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.940922526 +0000 UTC m=+10.131282430 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret") pod "global-pull-secret-syncer-s45x2" (UID: "9af1998d-6f11-4d4c-b45c-bcf73f88e1a1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:31.941543 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:31.941346    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:31.941543 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:31.941397    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs podName:2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.941379984 +0000 UTC m=+10.131739894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs") pod "network-metrics-daemon-gwxtd" (UID: "2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:32.042267 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:32.041623    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:32.042267 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:32.041809    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:32.042267 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:32.041830    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:32.042267 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:32.041842    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7z4bg for pod openshift-network-diagnostics/network-check-target-dqvff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:32.042267 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:32.041928    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg podName:582ab465-95af-4733-970f-8e3cee8ca49e nodeName:}" failed. No retries permitted until 2026-07-09 17:39:36.041899544 +0000 UTC m=+10.232259459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z4bg" (UniqueName: "kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg") pod "network-check-target-dqvff" (UID: "582ab465-95af-4733-970f-8e3cee8ca49e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:32.413728 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:32.412502    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:32.413728 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:32.412646    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:32.413728 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:32.413055    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:32.413728 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:32.413159    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:32.413728 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:32.413208    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:32.413728 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:32.413276    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:34.413133 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:34.412261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:34.413133 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:34.412415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:34.413133 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:34.412923    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:34.413133 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:34.413024    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:34.413967 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:34.413809    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:34.413967 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:34.413929    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:35.976720 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:35.976682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:35.977218 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:35.976739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:35.977218 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:35.976879    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:35.977218 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:35.976957    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs podName:2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:43.976932223 +0000 UTC m=+18.167292138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs") pod "network-metrics-daemon-gwxtd" (UID: "2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:35.977218 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:35.976866    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:35.977218 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:35.977029    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret podName:9af1998d-6f11-4d4c-b45c-bcf73f88e1a1 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:43.977008042 +0000 UTC m=+18.167367939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret") pod "global-pull-secret-syncer-s45x2" (UID: "9af1998d-6f11-4d4c-b45c-bcf73f88e1a1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:36.077568 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:36.077540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:36.077734 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:36.077685    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:36.077734 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:36.077702    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:36.077734 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:36.077711    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7z4bg for pod openshift-network-diagnostics/network-check-target-dqvff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:36.077959 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:36.077762    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg podName:582ab465-95af-4733-970f-8e3cee8ca49e nodeName:}" failed. No retries permitted until 2026-07-09 17:39:44.077748583 +0000 UTC m=+18.268108477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z4bg" (UniqueName: "kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg") pod "network-check-target-dqvff" (UID: "582ab465-95af-4733-970f-8e3cee8ca49e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:36.414100 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:36.413459    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:36.414100 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:36.413591    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:36.414100 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:36.413664    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:36.414100 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:36.413782    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:36.414100 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:36.413831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:36.414100 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:36.413941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:38.416380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:38.415925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:38.416380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:38.415940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:38.416380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:38.415940    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:38.416380 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:38.416062    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:38.416380 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:38.416164    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:38.416380 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:38.416242    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:39.471124 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:39.470908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" event={"ID":"09aaeda5a43a1bc9c803c4ce38e0e34d","Type":"ContainerStarted","Data":"d90f9c90faaf15c8e38a067bff8c3d407e8199c86a7a78822593e7b096f13e55"}
Jul 09 17:39:39.472300 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:39.472273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal" event={"ID":"89b829e494294382f2ba3cb0e5e2f900","Type":"ContainerStarted","Data":"0c07312b70f608773964322f095839db838420a28d37be618fdf51e136f0e174"}
Jul 09 17:39:39.507793 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:39.507751    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-44.ec2.internal" podStartSLOduration=11.507737053 podStartE2EDuration="11.507737053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:39.507528255 +0000 UTC m=+13.697888170" watchObservedRunningTime="2026-07-09 17:39:39.507737053 +0000 UTC m=+13.698096968"
Jul 09 17:39:40.412156 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:40.412121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:40.412315 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:40.412121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:40.412315 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:40.412251    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:40.412315 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:40.412131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:40.412315 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:40.412297    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:40.412529 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:40.412399    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:40.475530 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:40.475499    2577 generic.go:356] "Generic (PLEG): container finished" podID="09aaeda5a43a1bc9c803c4ce38e0e34d" containerID="d90f9c90faaf15c8e38a067bff8c3d407e8199c86a7a78822593e7b096f13e55" exitCode=0
Jul 09 17:39:40.475973 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:40.475577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" event={"ID":"09aaeda5a43a1bc9c803c4ce38e0e34d","Type":"ContainerDied","Data":"d90f9c90faaf15c8e38a067bff8c3d407e8199c86a7a78822593e7b096f13e55"}
Jul 09 17:39:42.412806 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:42.412773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:42.413250 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:42.412824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:42.413250 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:42.412780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:42.413250 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:42.412934    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:42.413250 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:42.413071    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:42.413250 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:42.413191    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:44.044081 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:44.044046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:44.044081 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:44.044088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:44.044591 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.044173    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:44.044591 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.044174    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:44.044591 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.044228    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret podName:9af1998d-6f11-4d4c-b45c-bcf73f88e1a1 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:00.044210814 +0000 UTC m=+34.234570708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret") pod "global-pull-secret-syncer-s45x2" (UID: "9af1998d-6f11-4d4c-b45c-bcf73f88e1a1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:39:44.044591 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.044246    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs podName:2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:00.044238317 +0000 UTC m=+34.234598210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs") pod "network-metrics-daemon-gwxtd" (UID: "2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:39:44.144531 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:44.144501    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:44.144678 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.144637    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:39:44.144678 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.144660    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:39:44.144678 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.144672    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7z4bg for pod openshift-network-diagnostics/network-check-target-dqvff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:44.144794 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.144719    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg podName:582ab465-95af-4733-970f-8e3cee8ca49e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:00.144705196 +0000 UTC m=+34.335065089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z4bg" (UniqueName: "kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg") pod "network-check-target-dqvff" (UID: "582ab465-95af-4733-970f-8e3cee8ca49e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:39:44.412615 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:44.412546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:44.412615 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:44.412563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:44.412615 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:44.412590    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:44.412888 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.412698    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:44.412888 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.412815    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:44.412990 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:44.412927    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:46.415169 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:46.415138    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:46.415169 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:46.415166    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:46.415622 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:46.415171    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:46.415622 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:46.415257    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:46.415622 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:46.415334    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:46.415622 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:46.415389    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:48.412050 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.411933    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:48.412580 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.411956    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:48.412580 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:48.412147    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:48.412580 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:48.412240    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:48.412580 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.411960    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:48.412580 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:48.412347    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:48.490470 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.490352    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:39:48.490785 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.490767    2577 generic.go:356] "Generic (PLEG): container finished" podID="8abf9808-a4c1-4b03-8952-af5a3f1b63a3" containerID="6b9c3b3dd2ad0402597cad97443f813c29c38fcccc30fd4529f5dce828bcad7f" exitCode=1
Jul 09 17:39:48.490870 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.490850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"cd2a7f52782c0712d09aa4cfe7184e6c2b7deda4b0fbbf28fbd2a416e401c663"}
Jul 09 17:39:48.490949 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.490933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"9f11af1e30466eb28ff3fef1cb4d4e81aff102578155ff714c1daa761efbb556"}
Jul 09 17:39:48.491004 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.490990    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerDied","Data":"6b9c3b3dd2ad0402597cad97443f813c29c38fcccc30fd4529f5dce828bcad7f"}
Jul 09 17:39:48.491054 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.491029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"3c248f274c5148dbdecc1f17ee646cf0fafc3ce5040890187341b9a455c27b29"}
Jul 09 17:39:48.492164 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.492148    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e562981-a609-4972-95ee-ca9b7095b4f5" containerID="e3e5346a824294426ddef0be9ea25bc37daaa0f0c5056c623fae8941c67ca7b3" exitCode=0
Jul 09 17:39:48.492235 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.492208    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerDied","Data":"e3e5346a824294426ddef0be9ea25bc37daaa0f0c5056c623fae8941c67ca7b3"}
Jul 09 17:39:48.493610 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.493561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8j9n9" event={"ID":"25c30946-998e-4d0d-8673-a898a8247bd7","Type":"ContainerStarted","Data":"e61ca9197cc7602002c89911caef0ff8cb897a162d691952748290bd42706544"}
Jul 09 17:39:48.495437 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.495407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" event={"ID":"09aaeda5a43a1bc9c803c4ce38e0e34d","Type":"ContainerStarted","Data":"0be9a22b531051774419dc4d1c29d16839b44cf60986080e94b2222b4ef72e9e"}
Jul 09 17:39:48.497006 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.496975    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m9ntl" event={"ID":"338f5cc2-493a-43df-8aa6-a1e4b609de1f","Type":"ContainerStarted","Data":"3cb8b3212c4b513bfb215dada7975d627301fd207efd40d4fd7b097142bbc0e3"}
Jul 09 17:39:48.498600 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.498572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm" event={"ID":"35527b65-4b17-4865-a944-d581cd153815","Type":"ContainerStarted","Data":"6fe8835373f259b6f1f4a4da4754c123e31cebeb9179eb086170b5a9db643682"}
Jul 09 17:39:48.500553 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.500535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fp6j9" event={"ID":"5e7c5c76-19c3-41c8-a4f8-56d455d51dfa","Type":"ContainerStarted","Data":"bf2681169aa01e34f31a91130d589724065f5bfc7a34418c2b44b95ca18e2aae"}
Jul 09 17:39:48.501940 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.501920    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lcjp2" event={"ID":"de5fad1c-2d42-4610-97f3-595c50f0158e","Type":"ContainerStarted","Data":"c32105ce328f76ea5348031d5b6d8880ed5000b4e39dd96129733b27778d579b"}
Jul 09 17:39:48.537328 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.537282    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fp6j9" podStartSLOduration=2.19268431 podStartE2EDuration="21.537262194s" totalImagesPullingTime="19.344577884s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:48.536722003 +0000 UTC m=+22.727081918" watchObservedRunningTime="2026-07-09 17:39:48.537262194 +0000 UTC m=+22.727622109"
Jul 09 17:39:48.552978 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.552940    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-44.ec2.internal" podStartSLOduration=20.55292694 podStartE2EDuration="20.55292694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:48.55272762 +0000 UTC m=+22.743087531" watchObservedRunningTime="2026-07-09 17:39:48.55292694 +0000 UTC m=+22.743286855"
Jul 09 17:39:48.568493 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.568453    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8j9n9" podStartSLOduration=11.459875468 podStartE2EDuration="21.568436152s" totalImagesPullingTime="10.108560684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:48.568335983 +0000 UTC m=+22.758695900" watchObservedRunningTime="2026-07-09 17:39:48.568436152 +0000 UTC m=+22.758796068"
Jul 09 17:39:48.584609 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.584560    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m9ntl" podStartSLOduration=11.565949479 podStartE2EDuration="21.584546706s" totalImagesPullingTime="10.018597227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:48.584165223 +0000 UTC m=+22.774525139" watchObservedRunningTime="2026-07-09 17:39:48.584546706 +0000 UTC m=+22.774906621"
Jul 09 17:39:48.604493 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:48.602817    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lcjp2" podStartSLOduration=11.551382609000001 podStartE2EDuration="21.602797849s" totalImagesPullingTime="10.05141524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:48.601895963 +0000 UTC m=+22.792255878" watchObservedRunningTime="2026-07-09 17:39:48.602797849 +0000 UTC m=+22.793157764"
Jul 09 17:39:49.505749 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.505716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ns9fg" event={"ID":"67f6d6ab-4dfc-432a-85e7-7838ac2dba43","Type":"ContainerStarted","Data":"fdf9485db206cf853de15bb100e00d49fa0b3265b13894b8dd83c39e3ad165fa"}
Jul 09 17:39:49.508292 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.508272    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:39:49.508577 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.508556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"411673c4f88a08f289176b87a29cf8a20fee7dcc9fd04e797eb625fb9f4f122b"}
Jul 09 17:39:49.508637 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.508615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"067b2d6694f7ce65662b1b3f82dd67292d2339071cf32e346bc0194d1ca3c386"}
Jul 09 17:39:49.509941 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.509913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tdqd5" event={"ID":"38d9e254-7406-42bf-ab9c-97ede1271222","Type":"ContainerStarted","Data":"1fa3f28fc31c8d3ed25f8a376fe5064a8a8d5787f31eefe68941c5af04065725"}
Jul 09 17:39:49.529702 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.529653    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ns9fg" podStartSLOduration=2.695534392 podStartE2EDuration="22.529635643s" totalImagesPullingTime="19.834101251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:49.529490263 +0000 UTC m=+23.719850180" watchObservedRunningTime="2026-07-09 17:39:49.529635643 +0000 UTC m=+23.719995558"
Jul 09 17:39:49.547543 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.547507    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tdqd5" podStartSLOduration=12.475336803 podStartE2EDuration="22.547494325s" totalImagesPullingTime="10.072157522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:49.547262478 +0000 UTC m=+23.737622395" watchObservedRunningTime="2026-07-09 17:39:49.547494325 +0000 UTC m=+23.737854260"
Jul 09 17:39:49.668791 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:49.668756    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 17:39:50.284539 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:50.284406    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T17:39:49.668775157Z","UUID":"df7443f5-397e-4f5c-b9b0-0de14b5a2232","Handler":null,"Name":"","Endpoint":""}
Jul 09 17:39:50.286053 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:50.286001    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 17:39:50.286053 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:50.286041    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 17:39:50.412466 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:50.412438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:50.412608 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:50.412438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:50.412608 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:50.412557    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:50.412726 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:50.412646    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:50.412726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:50.412709    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:50.412827 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:50.412790    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:50.513852 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:50.513785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm" event={"ID":"35527b65-4b17-4865-a944-d581cd153815","Type":"ContainerStarted","Data":"3a7e6ca0a85b8872386aabd87fcd91b77f946ae5484d940acd80f76f72cb3bae"}
Jul 09 17:39:51.519300 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:51.519146    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:39:51.519738 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:51.519713    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"11426aaf834265d92e56012041624a80a4388aacde29975684a9b9fa438c285a"}
Jul 09 17:39:52.412582 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:52.412549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:52.412804 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:52.412563    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:52.412804 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:52.412664    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:52.412804 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:52.412688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:52.412998 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:52.412805    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:52.412998 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:52.412913    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:52.770982 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:52.770960    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:52.771479 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:52.771462    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:53.527131 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.526982    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:39:53.527529 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.527505    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"c8e11c0de7b250c617d84a0daedecca329ac87352ad98b2bbe825e3d8584f451"}
Jul 09 17:39:53.527856 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.527812    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:53.527969 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.527933    2577 scope.go:117] "RemoveContainer" containerID="6b9c3b3dd2ad0402597cad97443f813c29c38fcccc30fd4529f5dce828bcad7f"
Jul 09 17:39:53.529384 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.529362    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e562981-a609-4972-95ee-ca9b7095b4f5" containerID="7ea7afb4b21c6256f77248b8547e8b4f314d58fa9bd6c6c97669254cf9dafcf9" exitCode=0
Jul 09 17:39:53.529471 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.529445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerDied","Data":"7ea7afb4b21c6256f77248b8547e8b4f314d58fa9bd6c6c97669254cf9dafcf9"}
Jul 09 17:39:53.531732 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.531690    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm" event={"ID":"35527b65-4b17-4865-a944-d581cd153815","Type":"ContainerStarted","Data":"d7752a79b582295e4ba81beb88256543cc7804b7d3b4ebfaadca11e506b244e7"}
Jul 09 17:39:53.531893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.531877    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:53.532423 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.532381    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lcjp2"
Jul 09 17:39:53.545222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.545202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:53.644127 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:53.644087    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v2rjm" podStartSLOduration=4.6724283060000005 podStartE2EDuration="26.644071222s" totalImagesPullingTime="21.971642916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:53.643133566 +0000 UTC m=+27.833493481" watchObservedRunningTime="2026-07-09 17:39:53.644071222 +0000 UTC m=+27.834431141"
Jul 09 17:39:54.412809 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.412749    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:54.413464 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.412753    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:54.413464 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.412898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:54.413464 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:54.412912    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:54.413464 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:54.412976    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:54.413464 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:54.413089    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:54.537148 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.537118    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:39:54.537588 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.537555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" event={"ID":"8abf9808-a4c1-4b03-8952-af5a3f1b63a3","Type":"ContainerStarted","Data":"0487a5744ed2202f9c28b530b575e684cab76302112b561ac10cb8393bfd6ab4"}
Jul 09 17:39:54.537762 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.537741    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:54.538105 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.538033    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:54.552820 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.552800    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:39:54.573449 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:54.573403    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck" podStartSLOduration=8.178041138 podStartE2EDuration="27.573384231s" totalImagesPullingTime="19.395343093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:54.573066505 +0000 UTC m=+28.763426438" watchObservedRunningTime="2026-07-09 17:39:54.573384231 +0000 UTC m=+28.763744146"
Jul 09 17:39:55.004058 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.003899    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s45x2"]
Jul 09 17:39:55.004203 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.004194    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:55.004299 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:55.004281    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:55.008819 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.008782    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dqvff"]
Jul 09 17:39:55.008958 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.008898    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:55.009016 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:55.008979    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:55.009530 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.009509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gwxtd"]
Jul 09 17:39:55.009640 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.009612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:55.009748 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:55.009722    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:55.541559 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.541532    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e562981-a609-4972-95ee-ca9b7095b4f5" containerID="c2fb26f52bfe6dc60d4b0957ad808ca2e0e8a5d115e16d909e4ee298335454e4" exitCode=0
Jul 09 17:39:55.541919 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:55.541582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerDied","Data":"c2fb26f52bfe6dc60d4b0957ad808ca2e0e8a5d115e16d909e4ee298335454e4"}
Jul 09 17:39:56.413169 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:56.413145    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:56.413370 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:56.413213    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:39:56.413370 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:56.413287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:56.413498 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:56.413405    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:56.413498 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:56.413458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:56.413591 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:56.413549    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:57.547793 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:57.547760    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e562981-a609-4972-95ee-ca9b7095b4f5" containerID="a8a58a43c0e762886df9c95b128744c16d8b0b1e7a94ad514263b3000d7bb0ca" exitCode=0
Jul 09 17:39:57.548496 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:57.547806    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerDied","Data":"a8a58a43c0e762886df9c95b128744c16d8b0b1e7a94ad514263b3000d7bb0ca"}
Jul 09 17:39:58.412487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:58.412455    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:39:58.412643 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:58.412456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:39:58.412643 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:58.412578    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:39:58.412643 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:39:58.412588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:39:58.412793 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:58.412651    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:39:58.412793 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:39:58.412742    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:40:00.071018 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:00.070986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:40:00.071577 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:00.071036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:40:00.071577 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.071154    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:00.071577 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.071160    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:00.071577 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.071214    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret podName:9af1998d-6f11-4d4c-b45c-bcf73f88e1a1 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:32.071196614 +0000 UTC m=+66.261556536 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret") pod "global-pull-secret-syncer-s45x2" (UID: "9af1998d-6f11-4d4c-b45c-bcf73f88e1a1") : object "kube-system"/"original-pull-secret" not registered
Jul 09 17:40:00.071577 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.071245    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs podName:2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:32.071238026 +0000 UTC m=+66.261597919 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs") pod "network-metrics-daemon-gwxtd" (UID: "2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 17:40:00.171619 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:00.171582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:40:00.171769 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.171726    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 17:40:00.171769 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.171754    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 17:40:00.171769 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.171769    2577 projected.go:196] Error preparing data for projected volume kube-api-access-7z4bg for pod openshift-network-diagnostics/network-check-target-dqvff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:00.171924 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.171829    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg podName:582ab465-95af-4733-970f-8e3cee8ca49e nodeName:}" failed. No retries permitted until 2026-07-09 17:40:32.17180928 +0000 UTC m=+66.362169187 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7z4bg" (UniqueName: "kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg") pod "network-check-target-dqvff" (UID: "582ab465-95af-4733-970f-8e3cee8ca49e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 17:40:00.412556 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:00.412433    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:40:00.412556 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:00.412458    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:40:00.412758 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:00.412565    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:40:00.412758 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.412578    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gwxtd" podUID="2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89"
Jul 09 17:40:00.412758 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.412660    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-s45x2" podUID="9af1998d-6f11-4d4c-b45c-bcf73f88e1a1"
Jul 09 17:40:00.412758 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:00.412726    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dqvff" podUID="582ab465-95af-4733-970f-8e3cee8ca49e"
Jul 09 17:40:01.175616 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.175387    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-44.ec2.internal" event="NodeReady"
Jul 09 17:40:01.176046 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.175757    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 17:40:01.244103 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.244069    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-82fgg"]
Jul 09 17:40:01.246674 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.246654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.249229 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.249209    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6cwhv"]
Jul 09 17:40:01.250024 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.249988    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 17:40:01.250024 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.250020    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dxrqg\""
Jul 09 17:40:01.250165 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.250092    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 17:40:01.251154 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.251127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.254263 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.254241    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 17:40:01.255407 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.255391    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 17:40:01.255885 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.255603    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 17:40:01.255958 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.255899    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2srhc\""
Jul 09 17:40:01.265166 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.265149    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82fgg"]
Jul 09 17:40:01.270668 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.270649    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6cwhv"]
Jul 09 17:40:01.279692 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.279673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.279792 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.279721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmds\" (UniqueName: \"kubernetes.io/projected/fa624b20-2ac3-494f-b333-36a45b55a991-kube-api-access-lxmds\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.279860 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.279816    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa624b20-2ac3-494f-b333-36a45b55a991-tmp-dir\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.279903 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.279868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gt6z\" (UniqueName: \"kubernetes.io/projected/f04e2cc9-36ef-482b-a971-b66cb8cfd2a5-kube-api-access-9gt6z\") pod \"ingress-canary-6cwhv\" (UID: \"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5\") " pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.279956 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.279907    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f04e2cc9-36ef-482b-a971-b66cb8cfd2a5-cert\") pod \"ingress-canary-6cwhv\" (UID: \"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5\") " pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.279956 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.279932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa624b20-2ac3-494f-b333-36a45b55a991-config-volume\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.380373 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.380265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmds\" (UniqueName: \"kubernetes.io/projected/fa624b20-2ac3-494f-b333-36a45b55a991-kube-api-access-lxmds\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.380373 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.380324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa624b20-2ac3-494f-b333-36a45b55a991-tmp-dir\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.380373 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.380349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gt6z\" (UniqueName: \"kubernetes.io/projected/f04e2cc9-36ef-482b-a971-b66cb8cfd2a5-kube-api-access-9gt6z\") pod \"ingress-canary-6cwhv\" (UID: \"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5\") " pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.380634 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.380381    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f04e2cc9-36ef-482b-a971-b66cb8cfd2a5-cert\") pod \"ingress-canary-6cwhv\" (UID: \"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5\") " pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.380634 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.380404    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa624b20-2ac3-494f-b333-36a45b55a991-config-volume\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.380634 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.380462    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.380634 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:01.380558    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:01.380634 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:01.380613    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls podName:fa624b20-2ac3-494f-b333-36a45b55a991 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:01.88059474 +0000 UTC m=+36.070954637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls") pod "dns-default-82fgg" (UID: "fa624b20-2ac3-494f-b333-36a45b55a991") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:01.380971 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.380636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa624b20-2ac3-494f-b333-36a45b55a991-tmp-dir\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.381040 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.381021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa624b20-2ac3-494f-b333-36a45b55a991-config-volume\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.384907 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.384884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f04e2cc9-36ef-482b-a971-b66cb8cfd2a5-cert\") pod \"ingress-canary-6cwhv\" (UID: \"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5\") " pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.391799 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.391773    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gt6z\" (UniqueName: \"kubernetes.io/projected/f04e2cc9-36ef-482b-a971-b66cb8cfd2a5-kube-api-access-9gt6z\") pod \"ingress-canary-6cwhv\" (UID: \"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5\") " pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.392325 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.392305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmds\" (UniqueName: \"kubernetes.io/projected/fa624b20-2ac3-494f-b333-36a45b55a991-kube-api-access-lxmds\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.567332 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.567302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6cwhv"
Jul 09 17:40:01.823495 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.823467    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6cwhv"]
Jul 09 17:40:01.827915 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:01.827883    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04e2cc9_36ef_482b_a971_b66cb8cfd2a5.slice/crio-c45e151f802e8fb2f9d0708d6a9b7b6038ebd16a7e7aa47218cc93b2a04a5566 WatchSource:0}: Error finding container c45e151f802e8fb2f9d0708d6a9b7b6038ebd16a7e7aa47218cc93b2a04a5566: Status 404 returned error can't find the container with id c45e151f802e8fb2f9d0708d6a9b7b6038ebd16a7e7aa47218cc93b2a04a5566
Jul 09 17:40:01.883276 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:01.883243    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:01.883414 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:01.883394    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:01.883472 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:01.883456    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls podName:fa624b20-2ac3-494f-b333-36a45b55a991 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:02.883438661 +0000 UTC m=+37.073798572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls") pod "dns-default-82fgg" (UID: "fa624b20-2ac3-494f-b333-36a45b55a991") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:02.412703 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.412532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:40:02.413048 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.412532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:40:02.413048 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.412583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:40:02.416232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.416202    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 17:40:02.416299 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.416240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 17:40:02.417527 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.417506    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 17:40:02.417607 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.417556    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d4pd\""
Jul 09 17:40:02.417607 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.417556    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wznvh\""
Jul 09 17:40:02.417718 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.417641    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 17:40:02.560820 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.560793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6cwhv" event={"ID":"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5","Type":"ContainerStarted","Data":"c45e151f802e8fb2f9d0708d6a9b7b6038ebd16a7e7aa47218cc93b2a04a5566"}
Jul 09 17:40:02.565768 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.565748    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e562981-a609-4972-95ee-ca9b7095b4f5" containerID="3adb9c34ffed5cfaca24db0aa00546ec5160cb6f2474f573742958df7c0d91a7" exitCode=0
Jul 09 17:40:02.565859 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.565776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerDied","Data":"3adb9c34ffed5cfaca24db0aa00546ec5160cb6f2474f573742958df7c0d91a7"}
Jul 09 17:40:02.888687 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:02.888654    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:02.888792 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:02.888776    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:02.888854 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:02.888823    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls podName:fa624b20-2ac3-494f-b333-36a45b55a991 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:04.888809599 +0000 UTC m=+39.079169492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls") pod "dns-default-82fgg" (UID: "fa624b20-2ac3-494f-b333-36a45b55a991") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:03.571929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:03.571891    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e562981-a609-4972-95ee-ca9b7095b4f5" containerID="bd7bce9d5d484e27024addc4e627f7a322b636f356078a62aa25478b01f4fb05" exitCode=0
Jul 09 17:40:03.572379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:03.571950    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerDied","Data":"bd7bce9d5d484e27024addc4e627f7a322b636f356078a62aa25478b01f4fb05"}
Jul 09 17:40:04.578548 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:04.578513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" event={"ID":"7e562981-a609-4972-95ee-ca9b7095b4f5","Type":"ContainerStarted","Data":"33c8984163bc45f288bbec80ef1254eadad149f49eeb13b37f6c9d9bb7f7300c"}
Jul 09 17:40:04.607598 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:04.607557    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t4m8f" podStartSLOduration=12.014043288 podStartE2EDuration="37.607541678s" totalImagesPullingTime="25.59349839s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:04.602936562 +0000 UTC m=+38.793296477" watchObservedRunningTime="2026-07-09 17:40:04.607541678 +0000 UTC m=+38.797901593"
Jul 09 17:40:04.902134 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:04.902064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:04.902259 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:04.902173    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:04.902259 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:04.902225    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls podName:fa624b20-2ac3-494f-b333-36a45b55a991 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.902210684 +0000 UTC m=+43.092570577 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls") pod "dns-default-82fgg" (UID: "fa624b20-2ac3-494f-b333-36a45b55a991") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:05.583202 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:05.583172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6cwhv" event={"ID":"f04e2cc9-36ef-482b-a971-b66cb8cfd2a5","Type":"ContainerStarted","Data":"65359bbc4eecb1e8c2995a63ed24f7b716d36b6b81f6a23e316da0377ce31c9d"}
Jul 09 17:40:05.601893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:05.601831    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6cwhv" podStartSLOduration=1.316773449 podStartE2EDuration="4.601816985s" totalImagesPullingTime="3.285043536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:05.600800919 +0000 UTC m=+39.791160833" watchObservedRunningTime="2026-07-09 17:40:05.601816985 +0000 UTC m=+39.792176897"
Jul 09 17:40:06.417078 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.417051    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5vv52"]
Jul 09 17:40:06.438031 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.438014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.439310 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.439286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vv52"]
Jul 09 17:40:06.441156 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.441140    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 17:40:06.441657 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.441636    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 17:40:06.441770 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.441694    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 17:40:06.441770 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.441703    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 17:40:06.442434 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.442420    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2r4b\""
Jul 09 17:40:06.513337 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.513316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.513449 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.513347    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5l5\" (UniqueName: \"kubernetes.io/projected/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-api-access-9z5l5\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.513517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.513455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-crio-socket\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.513517 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.513478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.513619 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.513560    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-data-volume\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.614313 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.614289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.614323    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5l5\" (UniqueName: \"kubernetes.io/projected/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-api-access-9z5l5\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.614392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-crio-socket\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.614420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:06.614429    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:06.614492    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:07.114473325 +0000 UTC m=+41.304833218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.614523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-data-volume\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:06.614573    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.614575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-crio-socket\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.614791 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:06.614640    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:07.114623219 +0000 UTC m=+41.304983114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:06.615107 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.614821    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-data-volume\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.640717 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.640690    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5l5\" (UniqueName: \"kubernetes.io/projected/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-api-access-9z5l5\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:06.768628 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.768602    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"]
Jul 09 17:40:06.789098 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.789077    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"]
Jul 09 17:40:06.789188 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.789176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"
Jul 09 17:40:06.793602 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.793583    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 17:40:06.793693 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.793584    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nmrf9\""
Jul 09 17:40:06.793693 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.793650    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 17:40:06.816160 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.816141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9zmd\" (UniqueName: \"kubernetes.io/projected/42f5ac83-fa33-43c3-a27c-7e5c195bf93b-kube-api-access-c9zmd\") pod \"migrator-7f59b54f6c-rxzlx\" (UID: \"42f5ac83-fa33-43c3-a27c-7e5c195bf93b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"
Jul 09 17:40:06.917023 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.917003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9zmd\" (UniqueName: \"kubernetes.io/projected/42f5ac83-fa33-43c3-a27c-7e5c195bf93b-kube-api-access-c9zmd\") pod \"migrator-7f59b54f6c-rxzlx\" (UID: \"42f5ac83-fa33-43c3-a27c-7e5c195bf93b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"
Jul 09 17:40:06.926363 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:06.926341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9zmd\" (UniqueName: \"kubernetes.io/projected/42f5ac83-fa33-43c3-a27c-7e5c195bf93b-kube-api-access-c9zmd\") pod \"migrator-7f59b54f6c-rxzlx\" (UID: \"42f5ac83-fa33-43c3-a27c-7e5c195bf93b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"
Jul 09 17:40:07.097183 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:07.097136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"
Jul 09 17:40:07.118691 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:07.118621    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:07.118775 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:07.118720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:07.118775 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:07.118740    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:07.118945 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:07.118797    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.118778731 +0000 UTC m=+42.309138629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:07.118945 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:07.118900    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:07.119056 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:07.118973    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.118954338 +0000 UTC m=+42.309314237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:07.235098 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:07.235073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx"]
Jul 09 17:40:07.237582 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:07.237558    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f5ac83_fa33_43c3_a27c_7e5c195bf93b.slice/crio-4df4d4c870c50bcb29b4ed4ca95e7ed6134c71a73641c16307ce70b19fdeead6 WatchSource:0}: Error finding container 4df4d4c870c50bcb29b4ed4ca95e7ed6134c71a73641c16307ce70b19fdeead6: Status 404 returned error can't find the container with id 4df4d4c870c50bcb29b4ed4ca95e7ed6134c71a73641c16307ce70b19fdeead6
Jul 09 17:40:07.588381 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:07.588360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx" event={"ID":"42f5ac83-fa33-43c3-a27c-7e5c195bf93b","Type":"ContainerStarted","Data":"4df4d4c870c50bcb29b4ed4ca95e7ed6134c71a73641c16307ce70b19fdeead6"}
Jul 09 17:40:07.801964 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:07.801943    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8j9n9_25c30946-998e-4d0d-8673-a898a8247bd7/dns-node-resolver/0.log"
Jul 09 17:40:08.125819 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:08.125784    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:08.125985 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:08.125917    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:08.126042 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:08.125976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:08.126042 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:08.125992    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:10.125972962 +0000 UTC m=+44.316332879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:08.126170 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:08.126065    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:08.126170 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:08.126102    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:10.126092441 +0000 UTC m=+44.316452336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:08.929720 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:08.929674    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:08.929999 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:08.929772    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 17:40:08.929999 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:08.929815    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls podName:fa624b20-2ac3-494f-b333-36a45b55a991 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:16.929799758 +0000 UTC m=+51.120159654 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls") pod "dns-default-82fgg" (UID: "fa624b20-2ac3-494f-b333-36a45b55a991") : secret "dns-default-metrics-tls" not found
Jul 09 17:40:08.997594 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:08.997575    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m9ntl_338f5cc2-493a-43df-8aa6-a1e4b609de1f/node-ca/0.log"
Jul 09 17:40:09.398102 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:09.398081    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6cwhv_f04e2cc9-36ef-482b-a971-b66cb8cfd2a5/serve-healthcheck-canary/0.log"
Jul 09 17:40:09.594603 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:09.594576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx" event={"ID":"42f5ac83-fa33-43c3-a27c-7e5c195bf93b","Type":"ContainerStarted","Data":"932426ed20132ac7af3ed1eced3be72dc4b3e8779de81c1b1ff749c88b7cfbc3"}
Jul 09 17:40:09.594696 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:09.594656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx" event={"ID":"42f5ac83-fa33-43c3-a27c-7e5c195bf93b","Type":"ContainerStarted","Data":"44252bd5f61b8ee3495a81d9a6c86aae02bfa77fa4ec83c097c7a02d08c8af0d"}
Jul 09 17:40:09.623763 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:09.623726    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-rxzlx" podStartSLOduration=2.184909064 podStartE2EDuration="3.623714335s" totalImagesPullingTime="1.438805271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:09.62315655 +0000 UTC m=+43.813516466" watchObservedRunningTime="2026-07-09 17:40:09.623714335 +0000 UTC m=+43.814074249"
Jul 09 17:40:10.136738 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:10.136713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:10.137035 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:10.136765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:10.137035 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:10.136827    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:10.137035 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:10.136871    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:10.137035 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:10.136901    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:14.136885281 +0000 UTC m=+48.327245175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:10.137035 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:10.136916    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:14.136909332 +0000 UTC m=+48.327269224 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:14.165342 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:14.165310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:14.165778 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:14.165369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:14.165778 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:14.165435    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 17:40:14.165778 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:14.165454    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:14.165778 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:14.165506    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:22.165486832 +0000 UTC m=+56.355846726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : configmap "kube-rbac-proxy" not found
Jul 09 17:40:14.165778 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:14.165527    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls podName:61d30fe0-6f1d-467e-9e00-bc5d16ecb399 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:22.165517627 +0000 UTC m=+56.355877523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5vv52" (UID: "61d30fe0-6f1d-467e-9e00-bc5d16ecb399") : secret "insights-runtime-extractor-tls" not found
Jul 09 17:40:16.980713 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:16.980684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:16.982883 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:16.982864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa624b20-2ac3-494f-b333-36a45b55a991-metrics-tls\") pod \"dns-default-82fgg\" (UID: \"fa624b20-2ac3-494f-b333-36a45b55a991\") " pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:17.160278 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:17.160253    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:17.289181 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:17.289154    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-82fgg"]
Jul 09 17:40:17.297207 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:17.297182    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa624b20_2ac3_494f_b333_36a45b55a991.slice/crio-07c02c3f927160c37a99dbe0a8cc6574d95748fdc548cccbe2abd4e2db5f37b9 WatchSource:0}: Error finding container 07c02c3f927160c37a99dbe0a8cc6574d95748fdc548cccbe2abd4e2db5f37b9: Status 404 returned error can't find the container with id 07c02c3f927160c37a99dbe0a8cc6574d95748fdc548cccbe2abd4e2db5f37b9
Jul 09 17:40:17.612618 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:17.612593    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82fgg" event={"ID":"fa624b20-2ac3-494f-b333-36a45b55a991","Type":"ContainerStarted","Data":"07c02c3f927160c37a99dbe0a8cc6574d95748fdc548cccbe2abd4e2db5f37b9"}
Jul 09 17:40:19.623644 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:19.623584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82fgg" event={"ID":"fa624b20-2ac3-494f-b333-36a45b55a991","Type":"ContainerStarted","Data":"c6e98e26e17816d750789d48e49e62d054bd4391f1a1ddfab45023ad5ba09b52"}
Jul 09 17:40:19.623989 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:19.623652    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-82fgg" event={"ID":"fa624b20-2ac3-494f-b333-36a45b55a991","Type":"ContainerStarted","Data":"d6961e707d5e1d0b369990a0fe2066c268dd94f6b9691d136f276aa44aeb201b"}
Jul 09 17:40:19.623989 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:19.623721    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:19.648245 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:19.648203    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-82fgg" podStartSLOduration=16.642653043 podStartE2EDuration="18.648191107s" totalImagesPullingTime="2.005538064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:19.646428401 +0000 UTC m=+53.836788316" watchObservedRunningTime="2026-07-09 17:40:19.648191107 +0000 UTC m=+53.838551021"
Jul 09 17:40:22.217273 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:22.217238    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:22.217808 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:22.217310    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:22.217808 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:22.217768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:22.219550 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:22.219529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/61d30fe0-6f1d-467e-9e00-bc5d16ecb399-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5vv52\" (UID: \"61d30fe0-6f1d-467e-9e00-bc5d16ecb399\") " pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:22.347210 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:22.347184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5vv52"
Jul 09 17:40:22.469756 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:22.469695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5vv52"]
Jul 09 17:40:22.472375 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:22.472353    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d30fe0_6f1d_467e_9e00_bc5d16ecb399.slice/crio-496e25cbc76789930a0441e2c2b23a46114532b3038737512b1913b825260fd6 WatchSource:0}: Error finding container 496e25cbc76789930a0441e2c2b23a46114532b3038737512b1913b825260fd6: Status 404 returned error can't find the container with id 496e25cbc76789930a0441e2c2b23a46114532b3038737512b1913b825260fd6
Jul 09 17:40:22.634221 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:22.634191    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv52" event={"ID":"61d30fe0-6f1d-467e-9e00-bc5d16ecb399","Type":"ContainerStarted","Data":"496e25cbc76789930a0441e2c2b23a46114532b3038737512b1913b825260fd6"}
Jul 09 17:40:26.555598 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:26.555574    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sh5ck"
Jul 09 17:40:26.650986 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:26.650955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv52" event={"ID":"61d30fe0-6f1d-467e-9e00-bc5d16ecb399","Type":"ContainerStarted","Data":"c424a4da789716f273725c38ccf97d1710df50e83a71f22011340a7f49c3d2e2"}
Jul 09 17:40:28.343667 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.343639    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"]
Jul 09 17:40:28.345361 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.345344    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"
Jul 09 17:40:28.349300 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.349278    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:40:28.349443 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.349428    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 09 17:40:28.350625 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.350606    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-jktfm\""
Jul 09 17:40:28.375826 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.375805    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"]
Jul 09 17:40:28.377561 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.377545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.398639 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.398621    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 17:40:28.398797 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.398784    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pxs6j\""
Jul 09 17:40:28.398953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.398930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 17:40:28.399031 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.399013    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 17:40:28.399172 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.399159    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 17:40:28.399782 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.399768    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"]
Jul 09 17:40:28.401278 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.401262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.406521 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.406500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 17:40:28.408458 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.408440    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"]
Jul 09 17:40:28.409353 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.409336    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"]
Jul 09 17:40:28.411777 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.411759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.421003 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:28.420980    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-141-44.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-44.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret"
Jul 09 17:40:28.422678 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:28.422654    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-141-44.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-44.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret"
Jul 09 17:40:28.422880 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:28.422863    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-141-44.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-44.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret"
Jul 09 17:40:28.423984 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:28.423962    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-141-44.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-141-44.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret"
Jul 09 17:40:28.432512 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.432494    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"]
Jul 09 17:40:28.449852 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.449807    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"]
Jul 09 17:40:28.461793 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.461773    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"]
Jul 09 17:40:28.461979 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.461957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r7bv\" (UniqueName: \"kubernetes.io/projected/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-kube-api-access-5r7bv\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.462065 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462029    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgvrw\" (UniqueName: \"kubernetes.io/projected/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-kube-api-access-pgvrw\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.462065 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4b8a7ac-23d6-46f4-a0e8-040ba4eef652-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw\" (UID: \"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.462164 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.462164 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462098    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-tmp\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.462164 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdfq8\" (UniqueName: \"kubernetes.io/projected/a4b8a7ac-23d6-46f4-a0e8-040ba4eef652-kube-api-access-pdfq8\") pod \"managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw\" (UID: \"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.462164 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.462326 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.462326 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ptk\" (UniqueName: \"kubernetes.io/projected/23ef0c45-984c-42cd-b9b1-761ade3987f9-kube-api-access-p8ptk\") pod \"volume-data-source-validator-78495597f6-255cj\" (UID: \"23ef0c45-984c-42cd-b9b1-761ade3987f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"
Jul 09 17:40:28.462406 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.462406 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.462406 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.462386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-ca\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.514203 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.514179    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5746b555d6-9wj98"]
Jul 09 17:40:28.516291 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.516270    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.520497 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.520474    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 17:40:28.522078 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.522055    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 17:40:28.522370 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.522308    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 17:40:28.522742 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.522722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5d6lq\""
Jul 09 17:40:28.532487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.532465    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 17:40:28.552122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.552096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5746b555d6-9wj98"]
Jul 09 17:40:28.563383 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563353    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc28d\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-kube-api-access-vc28d\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.563507 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-ca\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.563507 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563472    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4b8a7ac-23d6-46f4-a0e8-040ba4eef652-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw\" (UID: \"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.563507 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.563662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-trusted-ca\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.563662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-image-registry-private-configuration\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.563662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdfq8\" (UniqueName: \"kubernetes.io/projected/a4b8a7ac-23d6-46f4-a0e8-040ba4eef652-kube-api-access-pdfq8\") pod \"managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw\" (UID: \"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.563662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.563964 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.563964 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-installation-pull-secrets\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.563964 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ptk\" (UniqueName: \"kubernetes.io/projected/23ef0c45-984c-42cd-b9b1-761ade3987f9-kube-api-access-p8ptk\") pod \"volume-data-source-validator-78495597f6-255cj\" (UID: \"23ef0c45-984c-42cd-b9b1-761ade3987f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"
Jul 09 17:40:28.563964 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.563964 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-bound-sa-token\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.564226 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r7bv\" (UniqueName: \"kubernetes.io/projected/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-kube-api-access-5r7bv\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.564226 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.563996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgvrw\" (UniqueName: \"kubernetes.io/projected/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-kube-api-access-pgvrw\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.564226 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.564029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-tmp\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.564226 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.564062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/45edecce-bd90-4352-8f0e-0abb04721587-ca-trust-extracted\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.564226 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.564095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-registry-tls\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.564226 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.564125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.564226 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.564153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-registry-certificates\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.564743 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.564721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-tmp\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.564816 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.564793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-additional-service-ca\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.566465 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.566443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a4b8a7ac-23d6-46f4-a0e8-040ba4eef652-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw\" (UID: \"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.567138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.567118    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-klusterlet-config\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.599874 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.599795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r7bv\" (UniqueName: \"kubernetes.io/projected/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-kube-api-access-5r7bv\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:28.601373 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.601352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ptk\" (UniqueName: \"kubernetes.io/projected/23ef0c45-984c-42cd-b9b1-761ade3987f9-kube-api-access-p8ptk\") pod \"volume-data-source-validator-78495597f6-255cj\" (UID: \"23ef0c45-984c-42cd-b9b1-761ade3987f9\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"
Jul 09 17:40:28.602266 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.602208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdfq8\" (UniqueName: \"kubernetes.io/projected/a4b8a7ac-23d6-46f4-a0e8-040ba4eef652-kube-api-access-pdfq8\") pod \"managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw\" (UID: \"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.606079 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.606058    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgvrw\" (UniqueName: \"kubernetes.io/projected/3f2fd167-66ee-4a38-9cb0-02b097c2d2e8-kube-api-access-pgvrw\") pod \"klusterlet-addon-workmgr-74c79c5958-hqbcr\" (UID: \"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.622380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.622357    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5746b555d6-9wj98"]
Jul 09 17:40:28.622599 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:28.622573    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-vc28d registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5746b555d6-9wj98" podUID="45edecce-bd90-4352-8f0e-0abb04721587"
Jul 09 17:40:28.653997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.653958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"
Jul 09 17:40:28.660665 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.660641    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv52" event={"ID":"61d30fe0-6f1d-467e-9e00-bc5d16ecb399","Type":"ContainerStarted","Data":"eb1895abab0a23878368c0dba4b179e5c343446d663ca9bf215a11d69787bfb5"}
Jul 09 17:40:28.660765 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.660678    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/45edecce-bd90-4352-8f0e-0abb04721587-ca-trust-extracted\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-registry-tls\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-registry-certificates\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc28d\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-kube-api-access-vc28d\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665544    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-trusted-ca\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-image-registry-private-configuration\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665606    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-installation-pull-secrets\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-bound-sa-token\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.665953 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.665751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/45edecce-bd90-4352-8f0e-0abb04721587-ca-trust-extracted\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.667210 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.666793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-registry-certificates\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.667387 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.667366    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-trusted-ca\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.668609 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.668584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-image-registry-private-configuration\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.669294 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.669267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-installation-pull-secrets\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.671121 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.671100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-registry-tls\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.676487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.676457    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc28d\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-kube-api-access-vc28d\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.677344 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.677309    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-bound-sa-token\") pod \"image-registry-5746b555d6-9wj98\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") " pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.700596 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.700568    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"
Jul 09 17:40:28.702720 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.702445    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:28.711609 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.710403    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767415    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-bound-sa-token\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767507    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/45edecce-bd90-4352-8f0e-0abb04721587-ca-trust-extracted\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767554    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-registry-certificates\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-trusted-ca\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767634    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-image-registry-private-configuration\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc28d\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-kube-api-access-vc28d\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767688    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-installation-pull-secrets\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.767724    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-registry-tls\") pod \"45edecce-bd90-4352-8f0e-0abb04721587\" (UID: \"45edecce-bd90-4352-8f0e-0abb04721587\") "
Jul 09 17:40:28.768929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.768623    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45edecce-bd90-4352-8f0e-0abb04721587-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:40:28.769102 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.769066    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:40:28.769602 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.769308    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:40:28.769602 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.769507    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-registry-certificates\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.769602 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.769526    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/45edecce-bd90-4352-8f0e-0abb04721587-trusted-ca\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.769602 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.769542    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/45edecce-bd90-4352-8f0e-0abb04721587-ca-trust-extracted\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.772563 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.772515    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:40:28.774888 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.774816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:40:28.778350 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.778316    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:40:28.778983 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.778913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-kube-api-access-vc28d" (OuterVolumeSpecName: "kube-api-access-vc28d") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "kube-api-access-vc28d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:40:28.779132 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.779102    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "45edecce-bd90-4352-8f0e-0abb04721587" (UID: "45edecce-bd90-4352-8f0e-0abb04721587"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:40:28.815261 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.815233    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj"]
Jul 09 17:40:28.819231 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:28.819171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ef0c45_984c_42cd_b9b1_761ade3987f9.slice/crio-0f833b14fbd1bfeba544caa834685070c1feef617e42c13649d5465d5f2bd229 WatchSource:0}: Error finding container 0f833b14fbd1bfeba544caa834685070c1feef617e42c13649d5465d5f2bd229: Status 404 returned error can't find the container with id 0f833b14fbd1bfeba544caa834685070c1feef617e42c13649d5465d5f2bd229
Jul 09 17:40:28.866464 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.866380    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw"]
Jul 09 17:40:28.870410 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:28.870386    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4b8a7ac_23d6_46f4_a0e8_040ba4eef652.slice/crio-5d838d31f7c9bb9f90d021b38f0d6e6a26c2baf2e3772012ec1a573234276b66 WatchSource:0}: Error finding container 5d838d31f7c9bb9f90d021b38f0d6e6a26c2baf2e3772012ec1a573234276b66: Status 404 returned error can't find the container with id 5d838d31f7c9bb9f90d021b38f0d6e6a26c2baf2e3772012ec1a573234276b66
Jul 09 17:40:28.870522 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.870483    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-image-registry-private-configuration\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.870522 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.870507    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc28d\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-kube-api-access-vc28d\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.870522 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.870521    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/45edecce-bd90-4352-8f0e-0abb04721587-installation-pull-secrets\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.870691 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.870531    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-registry-tls\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.870691 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.870539    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/45edecce-bd90-4352-8f0e-0abb04721587-bound-sa-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:40:28.885138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:28.885102    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"]
Jul 09 17:40:28.888213 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:28.888185    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2fd167_66ee_4a38_9cb0_02b097c2d2e8.slice/crio-f073b6ca0aa95a27861a989151d95e582c9789fadbeb027e638a5c4244444160 WatchSource:0}: Error finding container f073b6ca0aa95a27861a989151d95e582c9789fadbeb027e638a5c4244444160: Status 404 returned error can't find the container with id f073b6ca0aa95a27861a989151d95e582c9789fadbeb027e638a5c4244444160
Jul 09 17:40:29.239284 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.239208    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 17:40:29.246904 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.246876    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:29.480312 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.480054    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 17:40:29.490111 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.489989    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:29.542206 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.542011    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 17:40:29.552171 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.552116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-ca\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:29.565245 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:29.565079    2577 secret.go:189] Couldn't get secret open-cluster-management-agent-addon/cluster-proxy-hub-kubeconfig: failed to sync secret cache: timed out waiting for the condition
Jul 09 17:40:29.565245 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:40:29.565151    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub-kubeconfig podName:51b5bba4-0f24-4ca6-be44-6e547bbf4a80 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:30.065129744 +0000 UTC m=+64.255489647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "hub-kubeconfig" (UniqueName: "kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub-kubeconfig") pod "cluster-proxy-proxy-agent-579d67c8d9-vj5hs" (UID: "51b5bba4-0f24-4ca6-be44-6e547bbf4a80") : failed to sync secret cache: timed out waiting for the condition
Jul 09 17:40:29.630236 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.630206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-82fgg"
Jul 09 17:40:29.668279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.668203    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr" event={"ID":"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8","Type":"ContainerStarted","Data":"f073b6ca0aa95a27861a989151d95e582c9789fadbeb027e638a5c4244444160"}
Jul 09 17:40:29.673469 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.673411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw" event={"ID":"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652","Type":"ContainerStarted","Data":"5d838d31f7c9bb9f90d021b38f0d6e6a26c2baf2e3772012ec1a573234276b66"}
Jul 09 17:40:29.675284 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.675099    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 17:40:29.681714 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.676116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5746b555d6-9wj98"
Jul 09 17:40:29.681714 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.681026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj" event={"ID":"23ef0c45-984c-42cd-b9b1-761ade3987f9","Type":"ContainerStarted","Data":"0f833b14fbd1bfeba544caa834685070c1feef617e42c13649d5465d5f2bd229"}
Jul 09 17:40:29.737742 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.737714    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5746b555d6-9wj98"]
Jul 09 17:40:29.752164 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:29.752141    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5746b555d6-9wj98"]
Jul 09 17:40:30.085492 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:30.084987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:30.095998 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:30.094504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/51b5bba4-0f24-4ca6-be44-6e547bbf4a80-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-579d67c8d9-vj5hs\" (UID: \"51b5bba4-0f24-4ca6-be44-6e547bbf4a80\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:30.219611 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:30.219579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"
Jul 09 17:40:30.418575 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:30.418493    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45edecce-bd90-4352-8f0e-0abb04721587" path="/var/lib/kubelet/pods/45edecce-bd90-4352-8f0e-0abb04721587/volumes"
Jul 09 17:40:32.102131 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.102098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:40:32.102527 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.102155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:40:32.105023 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.104987    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 17:40:32.105138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.104988    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 17:40:32.115013 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.114985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9af1998d-6f11-4d4c-b45c-bcf73f88e1a1-original-pull-secret\") pod \"global-pull-secret-syncer-s45x2\" (UID: \"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1\") " pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:40:32.115149 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.115049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89-metrics-certs\") pod \"network-metrics-daemon-gwxtd\" (UID: \"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89\") " pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:40:32.124874 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.124847    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wznvh\""
Jul 09 17:40:32.128122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.128101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-s45x2"
Jul 09 17:40:32.132243 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.132217    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gwxtd"
Jul 09 17:40:32.203209 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.203172    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:40:32.206356 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.206336    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 17:40:32.215770 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.215754    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 17:40:32.226100 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.226076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4bg\" (UniqueName: \"kubernetes.io/projected/582ab465-95af-4733-970f-8e3cee8ca49e-kube-api-access-7z4bg\") pod \"network-check-target-dqvff\" (UID: \"582ab465-95af-4733-970f-8e3cee8ca49e\") " pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:40:32.435259 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.435196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8d4pd\""
Jul 09 17:40:32.443326 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:32.443302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:40:34.530733 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.528999    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs"]
Jul 09 17:40:34.535999 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.535938    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dqvff"]
Jul 09 17:40:34.541068 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:34.541046    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582ab465_95af_4733_970f_8e3cee8ca49e.slice/crio-91eb9f208243960a38e444d40495bbef492b54cdd6c4c9f17afdebdcb535ed37 WatchSource:0}: Error finding container 91eb9f208243960a38e444d40495bbef492b54cdd6c4c9f17afdebdcb535ed37: Status 404 returned error can't find the container with id 91eb9f208243960a38e444d40495bbef492b54cdd6c4c9f17afdebdcb535ed37
Jul 09 17:40:34.695314 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.695274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr" event={"ID":"3f2fd167-66ee-4a38-9cb0-02b097c2d2e8","Type":"ContainerStarted","Data":"6de30dec6e784e071c8880f947a925e03a56db49191f1b154d3293ab80f8372c"}
Jul 09 17:40:34.695456 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.695432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:34.696854 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.696805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw" event={"ID":"a4b8a7ac-23d6-46f4-a0e8-040ba4eef652","Type":"ContainerStarted","Data":"1dad127e518e1dc4606a5dd508c731e898737e3ebb512a80102974e69d0acb93"}
Jul 09 17:40:34.697007 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.696990    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr"
Jul 09 17:40:34.697887 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.697858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs" event={"ID":"51b5bba4-0f24-4ca6-be44-6e547bbf4a80","Type":"ContainerStarted","Data":"b2c2958d194201670bf8b8f6c683a8d7d6feb493de998b6247a8658939e84aea"}
Jul 09 17:40:34.698812 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.698791    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dqvff" event={"ID":"582ab465-95af-4733-970f-8e3cee8ca49e","Type":"ContainerStarted","Data":"91eb9f208243960a38e444d40495bbef492b54cdd6c4c9f17afdebdcb535ed37"}
Jul 09 17:40:34.700560 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.700540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5vv52" event={"ID":"61d30fe0-6f1d-467e-9e00-bc5d16ecb399","Type":"ContainerStarted","Data":"8e5f312e73349bf3d289a654a4a9ba0ad95002ecbb036ebe52284bd5a2d0da01"}
Jul 09 17:40:34.701763 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.701743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj" event={"ID":"23ef0c45-984c-42cd-b9b1-761ade3987f9","Type":"ContainerStarted","Data":"55db04b6c200c47c91f04ba33c2272b32f1a9b5a74e515987adafcaf32d2ff31"}
Jul 09 17:40:34.729813 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.729732    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-74c79c5958-hqbcr" podStartSLOduration=1.273687555 podStartE2EDuration="6.72971777s" totalImagesPullingTime="5.456030215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:34.728930415 +0000 UTC m=+68.919290329" watchObservedRunningTime="2026-07-09 17:40:34.72971777 +0000 UTC m=+68.920077680"
Jul 09 17:40:34.761785 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.761721    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-s45x2"]
Jul 09 17:40:34.764492 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:34.764472    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af1998d_6f11_4d4c_b45c_bcf73f88e1a1.slice/crio-f43953f57890b60a88aba52fed06c7ed3d17d4f81ea66679d2ed73ae0e958576 WatchSource:0}: Error finding container f43953f57890b60a88aba52fed06c7ed3d17d4f81ea66679d2ed73ae0e958576: Status 404 returned error can't find the container with id f43953f57890b60a88aba52fed06c7ed3d17d4f81ea66679d2ed73ae0e958576
Jul 09 17:40:34.771341 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.771317    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gwxtd"]
Jul 09 17:40:34.775584 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:34.775564    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7e695d_dd11_48d5_a6aa_beaf9b1a6d89.slice/crio-69fbf12d6498773d85b394811a1f3cfa6b080a6fb664a963fbb837c454bd293c WatchSource:0}: Error finding container 69fbf12d6498773d85b394811a1f3cfa6b080a6fb664a963fbb837c454bd293c: Status 404 returned error can't find the container with id 69fbf12d6498773d85b394811a1f3cfa6b080a6fb664a963fbb837c454bd293c
Jul 09 17:40:34.818203 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.818161    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5vv52" podStartSLOduration=17.2428521 podStartE2EDuration="28.818148087s" totalImagesPullingTime="11.575295987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:40:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:34.793715447 +0000 UTC m=+68.984075362" watchObservedRunningTime="2026-07-09 17:40:34.818148087 +0000 UTC m=+69.008507980"
Jul 09 17:40:34.818282 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.818248    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-255cj" podStartSLOduration=1.358322482 podStartE2EDuration="6.818242538s" totalImagesPullingTime="5.459920056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:34.817796743 +0000 UTC m=+69.008156658" watchObservedRunningTime="2026-07-09 17:40:34.818242538 +0000 UTC m=+69.008602452"
Jul 09 17:40:34.847677 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:34.847635    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6f749ddcd6-cw6lw" podStartSLOduration=1.396403484 podStartE2EDuration="6.847623401s" totalImagesPullingTime="5.451219917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:34.847246332 +0000 UTC m=+69.037606247" watchObservedRunningTime="2026-07-09 17:40:34.847623401 +0000 UTC m=+69.037983316"
Jul 09 17:40:35.708681 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:35.708637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s45x2" event={"ID":"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1","Type":"ContainerStarted","Data":"f43953f57890b60a88aba52fed06c7ed3d17d4f81ea66679d2ed73ae0e958576"}
Jul 09 17:40:35.712200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:35.711916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwxtd" event={"ID":"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89","Type":"ContainerStarted","Data":"69fbf12d6498773d85b394811a1f3cfa6b080a6fb664a963fbb837c454bd293c"}
Jul 09 17:40:36.719868 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:36.719723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwxtd" event={"ID":"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89","Type":"ContainerStarted","Data":"4cd20d3f971348e2db9171dc18ce4f6b6ebc7ad5f0d1167fc3de51ad8b54195c"}
Jul 09 17:40:36.719868 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:36.719797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gwxtd" event={"ID":"2f7e695d-dd11-48d5-a6aa-beaf9b1a6d89","Type":"ContainerStarted","Data":"a84ff217b1821eed29dfed1a6facad5aa7357c96d3ec1905a7a7b063f6943580"}
Jul 09 17:40:36.738463 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:36.738149    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gwxtd" podStartSLOduration=68.37122374 podStartE2EDuration="1m9.738133617s" totalImagesPullingTime="1.366909877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:36.737943417 +0000 UTC m=+70.928303334" watchObservedRunningTime="2026-07-09 17:40:36.738133617 +0000 UTC m=+70.928493534"
Jul 09 17:40:40.736691 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.736614    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dqvff" event={"ID":"582ab465-95af-4733-970f-8e3cee8ca49e","Type":"ContainerStarted","Data":"354c03d0cd40cfc7e7da1801d43fee54880b9a6b85d7aa9d40b661a0bacc5d98"}
Jul 09 17:40:40.737073 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.736730    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:40:40.738175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.738153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-s45x2" event={"ID":"9af1998d-6f11-4d4c-b45c-bcf73f88e1a1","Type":"ContainerStarted","Data":"73c396cbae3249c88c0b5b07b18204be69039f8929f9057bbfa44634b68372ed"}
Jul 09 17:40:40.740005 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.739981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs" event={"ID":"51b5bba4-0f24-4ca6-be44-6e547bbf4a80","Type":"ContainerStarted","Data":"dea5d4ef88d6ab5ff940a1d7797076bd6a838f52934fe5f9c0bb6c7a46466e6f"}
Jul 09 17:40:40.740079 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.740018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs" event={"ID":"51b5bba4-0f24-4ca6-be44-6e547bbf4a80","Type":"ContainerStarted","Data":"b815a14784e6f0bc4a21eef1234ebde4e52600f1fc30b45f681c578e0e69b6f9"}
Jul 09 17:40:40.740079 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.740039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs" event={"ID":"51b5bba4-0f24-4ca6-be44-6e547bbf4a80","Type":"ContainerStarted","Data":"347eb30ce3ff6b1880d546f0b06ed0ade5de1bd3b80a98681983fb95b104901f"}
Jul 09 17:40:40.755578 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.755535    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dqvff" podStartSLOduration=67.849407836 podStartE2EDuration="1m13.755523004s" totalImagesPullingTime="5.906115168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.754224546 +0000 UTC m=+74.944584458" watchObservedRunningTime="2026-07-09 17:40:40.755523004 +0000 UTC m=+74.945882909"
Jul 09 17:40:40.775154 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.775102    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-s45x2" podStartSLOduration=68.07816306 podStartE2EDuration="1m13.77508664s" totalImagesPullingTime="5.69692358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.774987728 +0000 UTC m=+74.965347644" watchObservedRunningTime="2026-07-09 17:40:40.77508664 +0000 UTC m=+74.965446552"
Jul 09 17:40:40.804772 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:40.804726    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-579d67c8d9-vj5hs" podStartSLOduration=6.878762683 podStartE2EDuration="12.804710809s" totalImagesPullingTime="5.925948126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:40.803737452 +0000 UTC m=+74.994097367" watchObservedRunningTime="2026-07-09 17:40:40.804710809 +0000 UTC m=+74.995070725"
Jul 09 17:40:46.692114 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.692078    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wl64d"]
Jul 09 17:40:46.695060 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.695044    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.698165 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.698139    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 17:40:46.698295 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.698233    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 17:40:46.698641 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.698624    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 17:40:46.699371 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.699349    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 17:40:46.699476 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.699411    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tbd5w\""
Jul 09 17:40:46.699528 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.699489    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 17:40:46.700039 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.700022    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 17:40:46.812030 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-metrics-client-ca\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812036    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-textfile\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-tls\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812138 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-wtmp\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-root\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfhwz\" (UniqueName: \"kubernetes.io/projected/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-kube-api-access-pfhwz\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.812357 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.812274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-sys\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913304 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-tls\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913304 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913249    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-wtmp\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913304 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913272    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-root\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913304 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913312    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfhwz\" (UniqueName: \"kubernetes.io/projected/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-kube-api-access-pfhwz\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-sys\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-root\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-metrics-client-ca\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-wtmp\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-sys\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-textfile\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913809 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913738    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-textfile\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913875 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-accelerators-collector-config\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.913932 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.913916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-metrics-client-ca\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.915565 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.915543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-tls\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.915636 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.915587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:46.929268 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:46.929250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfhwz\" (UniqueName: \"kubernetes.io/projected/b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a-kube-api-access-pfhwz\") pod \"node-exporter-wl64d\" (UID: \"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a\") " pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:47.004286 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:47.004268    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wl64d"
Jul 09 17:40:47.014430 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:40:47.014411    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2358ae9_30ad_4bc9_b8da_d1c31d7bd70a.slice/crio-f4208d36cb500c73ac4921bf3aa53be4f039035634c9a9260e35beed1d79fff0 WatchSource:0}: Error finding container f4208d36cb500c73ac4921bf3aa53be4f039035634c9a9260e35beed1d79fff0: Status 404 returned error can't find the container with id f4208d36cb500c73ac4921bf3aa53be4f039035634c9a9260e35beed1d79fff0
Jul 09 17:40:47.764512 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:47.764478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wl64d" event={"ID":"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a","Type":"ContainerStarted","Data":"f4208d36cb500c73ac4921bf3aa53be4f039035634c9a9260e35beed1d79fff0"}
Jul 09 17:40:48.769316 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:48.769283    2577 generic.go:356] "Generic (PLEG): container finished" podID="b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a" containerID="cbdfaa3692dba138bddb46d9323950ad50c15bf2309ec98624deb313025a2a95" exitCode=0
Jul 09 17:40:48.769631 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:48.769326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wl64d" event={"ID":"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a","Type":"ContainerDied","Data":"cbdfaa3692dba138bddb46d9323950ad50c15bf2309ec98624deb313025a2a95"}
Jul 09 17:40:49.775104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:49.775065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wl64d" event={"ID":"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a","Type":"ContainerStarted","Data":"506f852e6c3dc4f7aa63f285da87355d265147f76caafce3e6d94daac1401081"}
Jul 09 17:40:49.775606 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:49.775151    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wl64d" event={"ID":"b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a","Type":"ContainerStarted","Data":"946a4bdafe610849a8405376b37dad0e1d632c866a49c041b7de1f5581079591"}
Jul 09 17:40:49.820044 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:40:49.820003    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wl64d" podStartSLOduration=2.9702967620000003 podStartE2EDuration="3.819988441s" totalImagesPullingTime="787.147395ms" totalInitContainerRuntime="62.544284ms" isStatefulPod=true podCreationTimestamp="2026-07-09 17:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:49.818995122 +0000 UTC m=+84.009355037" watchObservedRunningTime="2026-07-09 17:40:49.819988441 +0000 UTC m=+84.010348355"
Jul 09 17:41:11.745624 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:41:11.745592    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dqvff"
Jul 09 17:44:26.199222 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:44:26.199197    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:44:26.204483 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:44:26.204460    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 17:44:26.216478 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:44:26.216455    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:46:48.816017 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.815944    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7787c55b6-zt6xb"]
Jul 09 17:46:48.818003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.817983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:48.821144 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.821123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 09 17:46:48.821258 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.821241    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 17:46:48.822386 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.822366    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-kf2gt\""
Jul 09 17:46:48.822541 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.822393    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 17:46:48.837662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.837643    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7787c55b6-zt6xb"]
Jul 09 17:46:48.877709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.877690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2wh\" (UniqueName: \"kubernetes.io/projected/879849b1-cf75-46cd-9086-79a538a95ead-kube-api-access-xp2wh\") pod \"llmisvc-controller-manager-7787c55b6-zt6xb\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") " pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:48.877803 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.877721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert\") pod \"llmisvc-controller-manager-7787c55b6-zt6xb\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") " pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:48.978182 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.978161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert\") pod \"llmisvc-controller-manager-7787c55b6-zt6xb\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") " pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:48.978270 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.978208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2wh\" (UniqueName: \"kubernetes.io/projected/879849b1-cf75-46cd-9086-79a538a95ead-kube-api-access-xp2wh\") pod \"llmisvc-controller-manager-7787c55b6-zt6xb\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") " pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:48.978313 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:46:48.978290    2577 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 09 17:46:48.978352 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:46:48.978340    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert podName:879849b1-cf75-46cd-9086-79a538a95ead nodeName:}" failed. No retries permitted until 2026-07-09 17:46:49.478326271 +0000 UTC m=+443.668686164 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert") pod "llmisvc-controller-manager-7787c55b6-zt6xb" (UID: "879849b1-cf75-46cd-9086-79a538a95ead") : secret "llmisvc-webhook-server-cert" not found
Jul 09 17:46:48.992203 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:48.992176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2wh\" (UniqueName: \"kubernetes.io/projected/879849b1-cf75-46cd-9086-79a538a95ead-kube-api-access-xp2wh\") pod \"llmisvc-controller-manager-7787c55b6-zt6xb\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") " pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:49.481658 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:49.481634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert\") pod \"llmisvc-controller-manager-7787c55b6-zt6xb\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") " pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:49.483896 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:49.483879    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert\") pod \"llmisvc-controller-manager-7787c55b6-zt6xb\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") " pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:49.727455 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:49.727435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:49.849139 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:49.849111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7787c55b6-zt6xb"]
Jul 09 17:46:49.852060 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:46:49.852035    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod879849b1_cf75_46cd_9086_79a538a95ead.slice/crio-d7f7a06994eb70c916c66e7f78bd2f1662841bc68e28d124df578c8d28afdcc2 WatchSource:0}: Error finding container d7f7a06994eb70c916c66e7f78bd2f1662841bc68e28d124df578c8d28afdcc2: Status 404 returned error can't find the container with id d7f7a06994eb70c916c66e7f78bd2f1662841bc68e28d124df578c8d28afdcc2
Jul 09 17:46:49.853457 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:49.853441    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:46:49.883726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:49.883701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb" event={"ID":"879849b1-cf75-46cd-9086-79a538a95ead","Type":"ContainerStarted","Data":"d7f7a06994eb70c916c66e7f78bd2f1662841bc68e28d124df578c8d28afdcc2"}
Jul 09 17:46:53.903269 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:53.903235    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb" event={"ID":"879849b1-cf75-46cd-9086-79a538a95ead","Type":"ContainerStarted","Data":"06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db"}
Jul 09 17:46:53.903663 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:53.903372    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:46:53.926602 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:46:53.926559    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb" podStartSLOduration=2.892161696 podStartE2EDuration="5.926547245s" totalImagesPullingTime="3.034385549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:46:53.924817142 +0000 UTC m=+448.115177058" watchObservedRunningTime="2026-07-09 17:46:53.926547245 +0000 UTC m=+448.116907168"
Jul 09 17:47:24.908722 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:24.908695    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 17:47:39.757214 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.757181    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"]
Jul 09 17:47:39.760304 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.760286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:39.763072 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.763049    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nmvq9\""
Jul 09 17:47:39.763461 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.763447    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:47:39.777921 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.777902    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 09 17:47:39.786096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.786073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"]
Jul 09 17:47:39.899112 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.899089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e2c11e2-b668-41f4-b465-1d9cac4d3af3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dbfbm\" (UID: \"3e2c11e2-b668-41f4-b465-1d9cac4d3af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:39.899209 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.899139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmzqc\" (UniqueName: \"kubernetes.io/projected/3e2c11e2-b668-41f4-b465-1d9cac4d3af3-kube-api-access-nmzqc\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dbfbm\" (UID: \"3e2c11e2-b668-41f4-b465-1d9cac4d3af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:39.999501 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.999474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmzqc\" (UniqueName: \"kubernetes.io/projected/3e2c11e2-b668-41f4-b465-1d9cac4d3af3-kube-api-access-nmzqc\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dbfbm\" (UID: \"3e2c11e2-b668-41f4-b465-1d9cac4d3af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:39.999599 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.999518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e2c11e2-b668-41f4-b465-1d9cac4d3af3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dbfbm\" (UID: \"3e2c11e2-b668-41f4-b465-1d9cac4d3af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:39.999860 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:39.999818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e2c11e2-b668-41f4-b465-1d9cac4d3af3-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dbfbm\" (UID: \"3e2c11e2-b668-41f4-b465-1d9cac4d3af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:40.013459 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:40.013416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmzqc\" (UniqueName: \"kubernetes.io/projected/3e2c11e2-b668-41f4-b465-1d9cac4d3af3-kube-api-access-nmzqc\") pod \"cert-manager-operator-controller-manager-684f79c9b9-dbfbm\" (UID: \"3e2c11e2-b668-41f4-b465-1d9cac4d3af3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:40.069391 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:40.069369    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"
Jul 09 17:47:40.209217 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:40.209180    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm"]
Jul 09 17:47:40.213655 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:47:40.213627    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2c11e2_b668_41f4_b465_1d9cac4d3af3.slice/crio-704edf17bb0315cc3a1e1b687f6f881c40bf5b66ca46bc1f36d9d1a387330d5f WatchSource:0}: Error finding container 704edf17bb0315cc3a1e1b687f6f881c40bf5b66ca46bc1f36d9d1a387330d5f: Status 404 returned error can't find the container with id 704edf17bb0315cc3a1e1b687f6f881c40bf5b66ca46bc1f36d9d1a387330d5f
Jul 09 17:47:41.051542 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:41.051514    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm" event={"ID":"3e2c11e2-b668-41f4-b465-1d9cac4d3af3","Type":"ContainerStarted","Data":"704edf17bb0315cc3a1e1b687f6f881c40bf5b66ca46bc1f36d9d1a387330d5f"}
Jul 09 17:47:44.064654 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:44.064587    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm" event={"ID":"3e2c11e2-b668-41f4-b465-1d9cac4d3af3","Type":"ContainerStarted","Data":"c6adaa0685d0f757dd84b3ae20c1ecd150db1e2321199849b669ccfd98270db8"}
Jul 09 17:47:44.090199 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:47:44.090145    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-dbfbm" podStartSLOduration=1.540068314 podStartE2EDuration="5.09013018s" totalImagesPullingTime="3.550061866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:47:44.089015903 +0000 UTC m=+498.279375815" watchObservedRunningTime="2026-07-09 17:47:44.09013018 +0000 UTC m=+498.280490095"
Jul 09 17:48:00.920514 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:00.920484    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"]
Jul 09 17:48:00.922664 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:00.922647    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:00.925551 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:00.925525    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-tkcwc\""
Jul 09 17:48:00.926556 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:00.926534    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 17:48:00.926683 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:00.926575    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 17:48:00.937196 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:00.937175    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"]
Jul 09 17:48:01.035150 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.035127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6stn\" (UniqueName: \"kubernetes.io/projected/da0085d4-4998-4cfa-970c-265c787d1e74-kube-api-access-s6stn\") pod \"openshift-lws-operator-fd8ccff4c-tlsc4\" (UID: \"da0085d4-4998-4cfa-970c-265c787d1e74\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:01.035233 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.035156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da0085d4-4998-4cfa-970c-265c787d1e74-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tlsc4\" (UID: \"da0085d4-4998-4cfa-970c-265c787d1e74\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:01.135466 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.135437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6stn\" (UniqueName: \"kubernetes.io/projected/da0085d4-4998-4cfa-970c-265c787d1e74-kube-api-access-s6stn\") pod \"openshift-lws-operator-fd8ccff4c-tlsc4\" (UID: \"da0085d4-4998-4cfa-970c-265c787d1e74\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:01.135594 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.135482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da0085d4-4998-4cfa-970c-265c787d1e74-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tlsc4\" (UID: \"da0085d4-4998-4cfa-970c-265c787d1e74\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:01.135805 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.135789    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da0085d4-4998-4cfa-970c-265c787d1e74-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tlsc4\" (UID: \"da0085d4-4998-4cfa-970c-265c787d1e74\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:01.144494 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.144467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6stn\" (UniqueName: \"kubernetes.io/projected/da0085d4-4998-4cfa-970c-265c787d1e74-kube-api-access-s6stn\") pod \"openshift-lws-operator-fd8ccff4c-tlsc4\" (UID: \"da0085d4-4998-4cfa-970c-265c787d1e74\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:01.231598 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.231549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"
Jul 09 17:48:01.358629 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:01.358606    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4"]
Jul 09 17:48:01.361958 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:48:01.361929    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda0085d4_4998_4cfa_970c_265c787d1e74.slice/crio-a6fd79187e10ec8bc8578356893aff496f0275f91610deb5201ad637457d316a WatchSource:0}: Error finding container a6fd79187e10ec8bc8578356893aff496f0275f91610deb5201ad637457d316a: Status 404 returned error can't find the container with id a6fd79187e10ec8bc8578356893aff496f0275f91610deb5201ad637457d316a
Jul 09 17:48:02.127732 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:02.127701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4" event={"ID":"da0085d4-4998-4cfa-970c-265c787d1e74","Type":"ContainerStarted","Data":"a6fd79187e10ec8bc8578356893aff496f0275f91610deb5201ad637457d316a"}
Jul 09 17:48:05.141812 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:05.141776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4" event={"ID":"da0085d4-4998-4cfa-970c-265c787d1e74","Type":"ContainerStarted","Data":"fd9460161228d084209dde3887c4be99009d5efeef5728d68379d69d54f3a168"}
Jul 09 17:48:05.163524 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:05.163481    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tlsc4" podStartSLOduration=1.862492633 podStartE2EDuration="5.163466258s" totalImagesPullingTime="3.300973625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:48:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:48:05.162020727 +0000 UTC m=+519.352380642" watchObservedRunningTime="2026-07-09 17:48:05.163466258 +0000 UTC m=+519.353826172"
Jul 09 17:48:23.238120 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.238036    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"]
Jul 09 17:48:23.240538 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.240516    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.243774 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.243747    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-249p8\""
Jul 09 17:48:23.244732 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.244712    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 17:48:23.244824 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.244728    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 17:48:23.244824 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.244777    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 17:48:23.254270 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.254251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"]
Jul 09 17:48:23.379819 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.379791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xz2m\" (UniqueName: \"kubernetes.io/projected/cf762c13-445f-43bd-912e-096388217608-kube-api-access-4xz2m\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.379915 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.379828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.379915 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.379862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.379997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.379910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.379997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.379960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf762c13-445f-43bd-912e-096388217608-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.379997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.379991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.380084 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.380014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.380084 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.380050    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf762c13-445f-43bd-912e-096388217608-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.380084 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.380078    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf762c13-445f-43bd-912e-096388217608-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.480776 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.480893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.480893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf762c13-445f-43bd-912e-096388217608-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.480893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf762c13-445f-43bd-912e-096388217608-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.480893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480862    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xz2m\" (UniqueName: \"kubernetes.io/projected/cf762c13-445f-43bd-912e-096388217608-kube-api-access-4xz2m\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.480893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481160 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481160 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481160 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.480947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf762c13-445f-43bd-912e-096388217608-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481325 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.481192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481325 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.481260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481325 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.481299    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481480 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.481434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.481662 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.481643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf762c13-445f-43bd-912e-096388217608-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.483115 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.483097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf762c13-445f-43bd-912e-096388217608-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.483201 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.483122    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf762c13-445f-43bd-912e-096388217608-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.490231 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.490183    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf762c13-445f-43bd-912e-096388217608-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.490231 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.490199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xz2m\" (UniqueName: \"kubernetes.io/projected/cf762c13-445f-43bd-912e-096388217608-kube-api-access-4xz2m\") pod \"openshift-ai-inference-openshift-default-866675cbb8-cvzqk\" (UID: \"cf762c13-445f-43bd-912e-096388217608\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.550921 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.550896    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:23.681485 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:23.681298    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"]
Jul 09 17:48:23.683985 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:48:23.683951    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf762c13_445f_43bd_912e_096388217608.slice/crio-4143bb0544eba730c7b07ab5fddd8d0c45760f3634643bda7581028da1c117a8 WatchSource:0}: Error finding container 4143bb0544eba730c7b07ab5fddd8d0c45760f3634643bda7581028da1c117a8: Status 404 returned error can't find the container with id 4143bb0544eba730c7b07ab5fddd8d0c45760f3634643bda7581028da1c117a8
Jul 09 17:48:24.211543 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:24.211502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" event={"ID":"cf762c13-445f-43bd-912e-096388217608","Type":"ContainerStarted","Data":"4143bb0544eba730c7b07ab5fddd8d0c45760f3634643bda7581028da1c117a8"}
Jul 09 17:48:28.432120 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:28.432072    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:48:28.432439 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:28.432148    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:48:28.432439 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:28.432179    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:48:29.235751 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:29.235711    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" event={"ID":"cf762c13-445f-43bd-912e-096388217608","Type":"ContainerStarted","Data":"e189af422966ce7291b2351280e2408ded6482258d3aff7b8c8d725ab6696c13"}
Jul 09 17:48:29.258605 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:29.258519    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" podStartSLOduration=1.5126185749999999 podStartE2EDuration="6.258505227s" totalImagesPullingTime="4.745886652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:48:29.258091386 +0000 UTC m=+543.448451300" watchObservedRunningTime="2026-07-09 17:48:29.258505227 +0000 UTC m=+543.448865143"
Jul 09 17:48:29.551746 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:29.551723    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:29.556083 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:29.556060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:30.239891 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:30.239864    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:30.240559 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:30.240548    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:48:32.167578 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.167548    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"]
Jul 09 17:48:32.169549 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.169530    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.173673 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.173651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 17:48:32.174741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.174719    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 17:48:32.174941 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.174924    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 17:48:32.175019 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.175006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-8gxlc\""
Jul 09 17:48:32.181022 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.181003    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"]
Jul 09 17:48:32.339864 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.339823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/50aef266-a676-4053-9bcc-ffb04572b455-manager-config\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.339970 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.339872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf4rf\" (UniqueName: \"kubernetes.io/projected/50aef266-a676-4053-9bcc-ffb04572b455-kube-api-access-qf4rf\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.339970 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.339916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/50aef266-a676-4053-9bcc-ffb04572b455-metrics-cert\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.340111 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.340093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50aef266-a676-4053-9bcc-ffb04572b455-cert\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.441352 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.441290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50aef266-a676-4053-9bcc-ffb04572b455-cert\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.441352 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.441324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/50aef266-a676-4053-9bcc-ffb04572b455-manager-config\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.441352 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.441343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf4rf\" (UniqueName: \"kubernetes.io/projected/50aef266-a676-4053-9bcc-ffb04572b455-kube-api-access-qf4rf\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.441536 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.441364    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/50aef266-a676-4053-9bcc-ffb04572b455-metrics-cert\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.441926 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.441907    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/50aef266-a676-4053-9bcc-ffb04572b455-manager-config\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.443587 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.443568    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/50aef266-a676-4053-9bcc-ffb04572b455-metrics-cert\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.443780 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.443759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50aef266-a676-4053-9bcc-ffb04572b455-cert\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.455006 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.454984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf4rf\" (UniqueName: \"kubernetes.io/projected/50aef266-a676-4053-9bcc-ffb04572b455-kube-api-access-qf4rf\") pod \"lws-controller-manager-6cd758bd69-zwxvw\" (UID: \"50aef266-a676-4053-9bcc-ffb04572b455\") " pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.478989 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.478967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:32.617779 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:32.617718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"]
Jul 09 17:48:32.619816 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:48:32.619792    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50aef266_a676_4053_9bcc_ffb04572b455.slice/crio-785b945313f0250a3eb174a8edc1f7664bc391d7cddc792a0813dffafd754840 WatchSource:0}: Error finding container 785b945313f0250a3eb174a8edc1f7664bc391d7cddc792a0813dffafd754840: Status 404 returned error can't find the container with id 785b945313f0250a3eb174a8edc1f7664bc391d7cddc792a0813dffafd754840
Jul 09 17:48:33.252756 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:33.252717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw" event={"ID":"50aef266-a676-4053-9bcc-ffb04572b455","Type":"ContainerStarted","Data":"785b945313f0250a3eb174a8edc1f7664bc391d7cddc792a0813dffafd754840"}
Jul 09 17:48:34.257951 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:34.257917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw" event={"ID":"50aef266-a676-4053-9bcc-ffb04572b455","Type":"ContainerStarted","Data":"551e5e9280fdc0bfa1bb935c48239b5147a46bdd717bab00e6f10b6e6672b98b"}
Jul 09 17:48:34.258310 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:34.258025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:48:34.277678 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:34.277632    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw" podStartSLOduration=0.965965081 podStartE2EDuration="2.277618487s" totalImagesPullingTime="1.311653406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:48:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:48:34.276774182 +0000 UTC m=+548.467134100" watchObservedRunningTime="2026-07-09 17:48:34.277618487 +0000 UTC m=+548.467978402"
Jul 09 17:48:45.264480 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:48:45.264450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6cd758bd69-zwxvw"
Jul 09 17:49:26.220610 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:26.220584    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:49:26.236554 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:26.236533    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:49:43.591690 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.591651    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"]
Jul 09 17:49:43.594687 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.594671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:43.597695 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.597664    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 17:49:43.597862 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.597744    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6l6kt\""
Jul 09 17:49:43.598923 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.598902    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 17:49:43.598923 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.598916    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 17:49:43.605185 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.605163    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"]
Jul 09 17:49:43.680884 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.680829    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"]
Jul 09 17:49:43.710423 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.710397    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/998d60ee-5732-4798-83ba-70a4510a8917-config-file\") pod \"limitador-limitador-ff68ff5cb-5wssx\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:43.710555 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.710425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnlj\" (UniqueName: \"kubernetes.io/projected/998d60ee-5732-4798-83ba-70a4510a8917-kube-api-access-gsnlj\") pod \"limitador-limitador-ff68ff5cb-5wssx\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:43.811726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.811695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/998d60ee-5732-4798-83ba-70a4510a8917-config-file\") pod \"limitador-limitador-ff68ff5cb-5wssx\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:43.811726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.811727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnlj\" (UniqueName: \"kubernetes.io/projected/998d60ee-5732-4798-83ba-70a4510a8917-kube-api-access-gsnlj\") pod \"limitador-limitador-ff68ff5cb-5wssx\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:43.812279 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.812262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/998d60ee-5732-4798-83ba-70a4510a8917-config-file\") pod \"limitador-limitador-ff68ff5cb-5wssx\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:43.820225 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.820197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnlj\" (UniqueName: \"kubernetes.io/projected/998d60ee-5732-4798-83ba-70a4510a8917-kube-api-access-gsnlj\") pod \"limitador-limitador-ff68ff5cb-5wssx\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:43.905706 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:43.905654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:44.044134 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.044109    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"]
Jul 09 17:49:44.046699 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:49:44.046668    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998d60ee_5732_4798_83ba_70a4510a8917.slice/crio-0e6e84755af0ae301d98d757334fb77ac83f76b0a93186f56c0c2ad6dc4f3190 WatchSource:0}: Error finding container 0e6e84755af0ae301d98d757334fb77ac83f76b0a93186f56c0c2ad6dc4f3190: Status 404 returned error can't find the container with id 0e6e84755af0ae301d98d757334fb77ac83f76b0a93186f56c0c2ad6dc4f3190
Jul 09 17:49:44.524179 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.524153    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx" event={"ID":"998d60ee-5732-4798-83ba-70a4510a8917","Type":"ContainerStarted","Data":"0e6e84755af0ae301d98d757334fb77ac83f76b0a93186f56c0c2ad6dc4f3190"}
Jul 09 17:49:44.735073 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.735045    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-cg5dt"]
Jul 09 17:49:44.739177 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.739160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cg5dt"
Jul 09 17:49:44.741872 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.741848    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kxx5n\""
Jul 09 17:49:44.748055 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.748033    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cg5dt"]
Jul 09 17:49:44.820324 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.820263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp7r6\" (UniqueName: \"kubernetes.io/projected/24da7721-aa8d-4079-98bb-3272a4aca717-kube-api-access-tp7r6\") pod \"authorino-8df697c9f-cg5dt\" (UID: \"24da7721-aa8d-4079-98bb-3272a4aca717\") " pod="kuadrant-system/authorino-8df697c9f-cg5dt"
Jul 09 17:49:44.921108 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.921084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp7r6\" (UniqueName: \"kubernetes.io/projected/24da7721-aa8d-4079-98bb-3272a4aca717-kube-api-access-tp7r6\") pod \"authorino-8df697c9f-cg5dt\" (UID: \"24da7721-aa8d-4079-98bb-3272a4aca717\") " pod="kuadrant-system/authorino-8df697c9f-cg5dt"
Jul 09 17:49:44.933238 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:44.933211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp7r6\" (UniqueName: \"kubernetes.io/projected/24da7721-aa8d-4079-98bb-3272a4aca717-kube-api-access-tp7r6\") pod \"authorino-8df697c9f-cg5dt\" (UID: \"24da7721-aa8d-4079-98bb-3272a4aca717\") " pod="kuadrant-system/authorino-8df697c9f-cg5dt"
Jul 09 17:49:45.048216 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:45.048189    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cg5dt"
Jul 09 17:49:45.188552 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:45.188427    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cg5dt"]
Jul 09 17:49:45.191661 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:49:45.191614    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24da7721_aa8d_4079_98bb_3272a4aca717.slice/crio-61c6977fabe154b3555266a873b5526478d92b713bf7ff294319426e3c51790c WatchSource:0}: Error finding container 61c6977fabe154b3555266a873b5526478d92b713bf7ff294319426e3c51790c: Status 404 returned error can't find the container with id 61c6977fabe154b3555266a873b5526478d92b713bf7ff294319426e3c51790c
Jul 09 17:49:45.529213 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:45.529182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cg5dt" event={"ID":"24da7721-aa8d-4079-98bb-3272a4aca717","Type":"ContainerStarted","Data":"61c6977fabe154b3555266a873b5526478d92b713bf7ff294319426e3c51790c"}
Jul 09 17:49:50.553666 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:50.553627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx" event={"ID":"998d60ee-5732-4798-83ba-70a4510a8917","Type":"ContainerStarted","Data":"eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a"}
Jul 09 17:49:50.554075 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:50.553701    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:49:50.555039 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:50.555020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cg5dt" event={"ID":"24da7721-aa8d-4079-98bb-3272a4aca717","Type":"ContainerStarted","Data":"288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1"}
Jul 09 17:49:50.574122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:50.574081    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx" podStartSLOduration=1.9662920480000001 podStartE2EDuration="7.574068292s" totalImagesPullingTime="5.607776244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:49:50.572343909 +0000 UTC m=+624.762703824" watchObservedRunningTime="2026-07-09 17:49:50.574068292 +0000 UTC m=+624.764428207"
Jul 09 17:49:50.588122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:49:50.588083    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-cg5dt" podStartSLOduration=2.181334293 podStartE2EDuration="6.588071029s" totalImagesPullingTime="4.406736736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:49:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:49:50.587218248 +0000 UTC m=+624.777578167" watchObservedRunningTime="2026-07-09 17:49:50.588071029 +0000 UTC m=+624.778430943"
Jul 09 17:50:00.561143 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:00.561105    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"]
Jul 09 17:50:00.561580 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:00.561299    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx" podUID="998d60ee-5732-4798-83ba-70a4510a8917" containerName="limitador" containerID="cri-o://eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a" gracePeriod=30
Jul 09 17:50:00.563521 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:00.563501    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:50:01.105940 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.105920    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:50:01.233218 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.233161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsnlj\" (UniqueName: \"kubernetes.io/projected/998d60ee-5732-4798-83ba-70a4510a8917-kube-api-access-gsnlj\") pod \"998d60ee-5732-4798-83ba-70a4510a8917\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") "
Jul 09 17:50:01.233218 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.233217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/998d60ee-5732-4798-83ba-70a4510a8917-config-file\") pod \"998d60ee-5732-4798-83ba-70a4510a8917\" (UID: \"998d60ee-5732-4798-83ba-70a4510a8917\") "
Jul 09 17:50:01.233547 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.233528    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998d60ee-5732-4798-83ba-70a4510a8917-config-file" (OuterVolumeSpecName: "config-file") pod "998d60ee-5732-4798-83ba-70a4510a8917" (UID: "998d60ee-5732-4798-83ba-70a4510a8917"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:50:01.235308 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.235288    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998d60ee-5732-4798-83ba-70a4510a8917-kube-api-access-gsnlj" (OuterVolumeSpecName: "kube-api-access-gsnlj") pod "998d60ee-5732-4798-83ba-70a4510a8917" (UID: "998d60ee-5732-4798-83ba-70a4510a8917"). InnerVolumeSpecName "kube-api-access-gsnlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:50:01.333921 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.333889    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/998d60ee-5732-4798-83ba-70a4510a8917-config-file\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:50:01.333921 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.333919    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsnlj\" (UniqueName: \"kubernetes.io/projected/998d60ee-5732-4798-83ba-70a4510a8917-kube-api-access-gsnlj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:50:01.598235 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.598205    2577 generic.go:356] "Generic (PLEG): container finished" podID="998d60ee-5732-4798-83ba-70a4510a8917" containerID="eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a" exitCode=0
Jul 09 17:50:01.598620 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.598254    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"
Jul 09 17:50:01.598620 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.598295    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx" event={"ID":"998d60ee-5732-4798-83ba-70a4510a8917","Type":"ContainerDied","Data":"eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a"}
Jul 09 17:50:01.598620 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.598371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-5wssx" event={"ID":"998d60ee-5732-4798-83ba-70a4510a8917","Type":"ContainerDied","Data":"0e6e84755af0ae301d98d757334fb77ac83f76b0a93186f56c0c2ad6dc4f3190"}
Jul 09 17:50:01.598620 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.598403    2577 scope.go:117] "RemoveContainer" containerID="eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a"
Jul 09 17:50:01.606927 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.606911    2577 scope.go:117] "RemoveContainer" containerID="eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a"
Jul 09 17:50:01.607168 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:50:01.607142    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a\": container with ID starting with eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a not found: ID does not exist" containerID="eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a"
Jul 09 17:50:01.607238 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.607182    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a"} err="failed to get container status \"eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a\": rpc error: code = NotFound desc = could not find container \"eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a\": container with ID starting with eba12c11434f244b1c67a06e1e6c683cc0d59034828df8437e37e484ffab7c4a not found: ID does not exist"
Jul 09 17:50:01.622726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.622706    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"]
Jul 09 17:50:01.627644 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:01.627624    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-5wssx"]
Jul 09 17:50:02.416880 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:02.416829    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="998d60ee-5732-4798-83ba-70a4510a8917" path="/var/lib/kubelet/pods/998d60ee-5732-4798-83ba-70a4510a8917/volumes"
Jul 09 17:50:13.100439 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.100407    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cg5dt"]
Jul 09 17:50:13.100890 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.100567    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-cg5dt" podUID="24da7721-aa8d-4079-98bb-3272a4aca717" containerName="authorino" containerID="cri-o://288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1" gracePeriod=30
Jul 09 17:50:13.342791 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.342769    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cg5dt"
Jul 09 17:50:13.411367 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.411309    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp7r6\" (UniqueName: \"kubernetes.io/projected/24da7721-aa8d-4079-98bb-3272a4aca717-kube-api-access-tp7r6\") pod \"24da7721-aa8d-4079-98bb-3272a4aca717\" (UID: \"24da7721-aa8d-4079-98bb-3272a4aca717\") "
Jul 09 17:50:13.413238 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.413212    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24da7721-aa8d-4079-98bb-3272a4aca717-kube-api-access-tp7r6" (OuterVolumeSpecName: "kube-api-access-tp7r6") pod "24da7721-aa8d-4079-98bb-3272a4aca717" (UID: "24da7721-aa8d-4079-98bb-3272a4aca717"). InnerVolumeSpecName "kube-api-access-tp7r6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:50:13.512578 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.512558    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp7r6\" (UniqueName: \"kubernetes.io/projected/24da7721-aa8d-4079-98bb-3272a4aca717-kube-api-access-tp7r6\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:50:13.645211 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.645179    2577 generic.go:356] "Generic (PLEG): container finished" podID="24da7721-aa8d-4079-98bb-3272a4aca717" containerID="288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1" exitCode=0
Jul 09 17:50:13.645336 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.645214    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-cg5dt"
Jul 09 17:50:13.645336 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.645219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cg5dt" event={"ID":"24da7721-aa8d-4079-98bb-3272a4aca717","Type":"ContainerDied","Data":"288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1"}
Jul 09 17:50:13.645336 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.645250    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-cg5dt" event={"ID":"24da7721-aa8d-4079-98bb-3272a4aca717","Type":"ContainerDied","Data":"61c6977fabe154b3555266a873b5526478d92b713bf7ff294319426e3c51790c"}
Jul 09 17:50:13.645336 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.645270    2577 scope.go:117] "RemoveContainer" containerID="288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1"
Jul 09 17:50:13.653402 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.653386    2577 scope.go:117] "RemoveContainer" containerID="288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1"
Jul 09 17:50:13.653655 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:50:13.653635    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1\": container with ID starting with 288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1 not found: ID does not exist" containerID="288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1"
Jul 09 17:50:13.653703 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.653662    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1"} err="failed to get container status \"288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1\": rpc error: code = NotFound desc = could not find container \"288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1\": container with ID starting with 288ed66dc4578cb1702b8978569926d26b58b198594897522971cf381dd1c8b1 not found: ID does not exist"
Jul 09 17:50:13.667001 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.666954    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cg5dt"]
Jul 09 17:50:13.669256 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:13.669238    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-cg5dt"]
Jul 09 17:50:14.417509 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:14.417479    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24da7721-aa8d-4079-98bb-3272a4aca717" path="/var/lib/kubelet/pods/24da7721-aa8d-4079-98bb-3272a4aca717/volumes"
Jul 09 17:50:51.176949 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.176917    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-vl2hg"]
Jul 09 17:50:51.177487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.177204    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="998d60ee-5732-4798-83ba-70a4510a8917" containerName="limitador"
Jul 09 17:50:51.177487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.177218    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="998d60ee-5732-4798-83ba-70a4510a8917" containerName="limitador"
Jul 09 17:50:51.177487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.177233    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24da7721-aa8d-4079-98bb-3272a4aca717" containerName="authorino"
Jul 09 17:50:51.177487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.177238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24da7721-aa8d-4079-98bb-3272a4aca717" containerName="authorino"
Jul 09 17:50:51.177487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.177287    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24da7721-aa8d-4079-98bb-3272a4aca717" containerName="authorino"
Jul 09 17:50:51.177487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.177295    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="998d60ee-5732-4798-83ba-70a4510a8917" containerName="limitador"
Jul 09 17:50:51.180088 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.180071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.183327 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.183303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 17:50:51.183784 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.183768    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-f7v6k\""
Jul 09 17:50:51.196620 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.196601    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-vl2hg"]
Jul 09 17:50:51.202657 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.202636    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-77l6h"]
Jul 09 17:50:51.205558 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.205542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:51.208734 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.208715    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 09 17:50:51.208826 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.208720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-xr48t\""
Jul 09 17:50:51.221409 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.221390    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-77l6h"]
Jul 09 17:50:51.363578 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.363554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a8c86d-d109-490e-abe9-9e4d22a3caf5-tls-certs\") pod \"model-serving-api-fd65d7d6-vl2hg\" (UID: \"80a8c86d-d109-490e-abe9-9e4d22a3caf5\") " pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.363698 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.363582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgv9\" (UniqueName: \"kubernetes.io/projected/80a8c86d-d109-490e-abe9-9e4d22a3caf5-kube-api-access-6rgv9\") pod \"model-serving-api-fd65d7d6-vl2hg\" (UID: \"80a8c86d-d109-490e-abe9-9e4d22a3caf5\") " pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.363698 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.363605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n49bz\" (UniqueName: \"kubernetes.io/projected/2c4791d2-4d43-4b09-8688-49742c15025b-kube-api-access-n49bz\") pod \"odh-model-controller-84b54b7f97-77l6h\" (UID: \"2c4791d2-4d43-4b09-8688-49742c15025b\") " pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:51.363777 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.363710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c4791d2-4d43-4b09-8688-49742c15025b-cert\") pod \"odh-model-controller-84b54b7f97-77l6h\" (UID: \"2c4791d2-4d43-4b09-8688-49742c15025b\") " pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:51.464457 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.464407    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c4791d2-4d43-4b09-8688-49742c15025b-cert\") pod \"odh-model-controller-84b54b7f97-77l6h\" (UID: \"2c4791d2-4d43-4b09-8688-49742c15025b\") " pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:51.464457 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.464443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a8c86d-d109-490e-abe9-9e4d22a3caf5-tls-certs\") pod \"model-serving-api-fd65d7d6-vl2hg\" (UID: \"80a8c86d-d109-490e-abe9-9e4d22a3caf5\") " pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.464578 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.464461    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgv9\" (UniqueName: \"kubernetes.io/projected/80a8c86d-d109-490e-abe9-9e4d22a3caf5-kube-api-access-6rgv9\") pod \"model-serving-api-fd65d7d6-vl2hg\" (UID: \"80a8c86d-d109-490e-abe9-9e4d22a3caf5\") " pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.464578 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.464497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n49bz\" (UniqueName: \"kubernetes.io/projected/2c4791d2-4d43-4b09-8688-49742c15025b-kube-api-access-n49bz\") pod \"odh-model-controller-84b54b7f97-77l6h\" (UID: \"2c4791d2-4d43-4b09-8688-49742c15025b\") " pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:51.464578 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:50:51.464554    2577 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 17:50:51.464668 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:50:51.464617    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c4791d2-4d43-4b09-8688-49742c15025b-cert podName:2c4791d2-4d43-4b09-8688-49742c15025b nodeName:}" failed. No retries permitted until 2026-07-09 17:50:51.964597413 +0000 UTC m=+686.154957308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/2c4791d2-4d43-4b09-8688-49742c15025b-cert") pod "odh-model-controller-84b54b7f97-77l6h" (UID: "2c4791d2-4d43-4b09-8688-49742c15025b") : secret "odh-model-controller-webhook-cert" not found
Jul 09 17:50:51.466711 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.466693    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80a8c86d-d109-490e-abe9-9e4d22a3caf5-tls-certs\") pod \"model-serving-api-fd65d7d6-vl2hg\" (UID: \"80a8c86d-d109-490e-abe9-9e4d22a3caf5\") " pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.475770 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.475742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n49bz\" (UniqueName: \"kubernetes.io/projected/2c4791d2-4d43-4b09-8688-49742c15025b-kube-api-access-n49bz\") pod \"odh-model-controller-84b54b7f97-77l6h\" (UID: \"2c4791d2-4d43-4b09-8688-49742c15025b\") " pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:51.475861 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.475823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgv9\" (UniqueName: \"kubernetes.io/projected/80a8c86d-d109-490e-abe9-9e4d22a3caf5-kube-api-access-6rgv9\") pod \"model-serving-api-fd65d7d6-vl2hg\" (UID: \"80a8c86d-d109-490e-abe9-9e4d22a3caf5\") " pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.489694 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.489675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:51.617613 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.617583    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-vl2hg"]
Jul 09 17:50:51.619674 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:50:51.619650    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a8c86d_d109_490e_abe9_9e4d22a3caf5.slice/crio-caffccf2854777bf358ecd33f91f04a839c872b904eac070d5fe83638f6c83f9 WatchSource:0}: Error finding container caffccf2854777bf358ecd33f91f04a839c872b904eac070d5fe83638f6c83f9: Status 404 returned error can't find the container with id caffccf2854777bf358ecd33f91f04a839c872b904eac070d5fe83638f6c83f9
Jul 09 17:50:51.787519 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.787492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-vl2hg" event={"ID":"80a8c86d-d109-490e-abe9-9e4d22a3caf5","Type":"ContainerStarted","Data":"caffccf2854777bf358ecd33f91f04a839c872b904eac070d5fe83638f6c83f9"}
Jul 09 17:50:51.969554 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.969528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c4791d2-4d43-4b09-8688-49742c15025b-cert\") pod \"odh-model-controller-84b54b7f97-77l6h\" (UID: \"2c4791d2-4d43-4b09-8688-49742c15025b\") " pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:51.971782 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:51.971762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2c4791d2-4d43-4b09-8688-49742c15025b-cert\") pod \"odh-model-controller-84b54b7f97-77l6h\" (UID: \"2c4791d2-4d43-4b09-8688-49742c15025b\") " pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:52.114970 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:52.114921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:52.244543 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:52.244520    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-77l6h"]
Jul 09 17:50:52.246236 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:50:52.246209    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4791d2_4d43_4b09_8688_49742c15025b.slice/crio-34d677c318e02e5d112ce850470a7b2a31db0497d1aeadade59aa56b0413efb5 WatchSource:0}: Error finding container 34d677c318e02e5d112ce850470a7b2a31db0497d1aeadade59aa56b0413efb5: Status 404 returned error can't find the container with id 34d677c318e02e5d112ce850470a7b2a31db0497d1aeadade59aa56b0413efb5
Jul 09 17:50:52.794902 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:52.794858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-77l6h" event={"ID":"2c4791d2-4d43-4b09-8688-49742c15025b","Type":"ContainerStarted","Data":"34d677c318e02e5d112ce850470a7b2a31db0497d1aeadade59aa56b0413efb5"}
Jul 09 17:50:53.800878 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:53.800825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-vl2hg" event={"ID":"80a8c86d-d109-490e-abe9-9e4d22a3caf5","Type":"ContainerStarted","Data":"263f36740c7689d43cdf9457c5963ad319247dc28b5b48c1cd5f065b6b26bbc5"}
Jul 09 17:50:53.801276 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:53.800925    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:50:53.824261 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:53.824211    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-vl2hg" podStartSLOduration=1.094078503 podStartE2EDuration="2.824192858s" totalImagesPullingTime="1.730114355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:50:53.822245918 +0000 UTC m=+688.012605834" watchObservedRunningTime="2026-07-09 17:50:53.824192858 +0000 UTC m=+688.014552772"
Jul 09 17:50:55.812146 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:55.812112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-77l6h" event={"ID":"2c4791d2-4d43-4b09-8688-49742c15025b","Type":"ContainerStarted","Data":"e0a72d802ce3de57b2a5ac8eea1057da6964a11aeab4d61acf686a214d8bbca0"}
Jul 09 17:50:55.812547 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:55.812239    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:50:55.838514 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:50:55.838465    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-77l6h" podStartSLOduration=2.008318882 podStartE2EDuration="4.838451289s" totalImagesPullingTime="2.830132407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:50:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:50:55.836723116 +0000 UTC m=+690.027083031" watchObservedRunningTime="2026-07-09 17:50:55.838451289 +0000 UTC m=+690.028811204"
Jul 09 17:51:04.810620 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:04.810596    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-vl2hg"
Jul 09 17:51:06.817199 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:06.817168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-77l6h"
Jul 09 17:51:07.724039 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.724013    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-5n4x5"]
Jul 09 17:51:07.727096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.727080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5n4x5"
Jul 09 17:51:07.729741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.729720    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 17:51:07.729862 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.729721    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-nrr6p\""
Jul 09 17:51:07.736576 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.736552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5n4x5"]
Jul 09 17:51:07.883113 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.883089    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gfg4\" (UniqueName: \"kubernetes.io/projected/bdc3e083-903b-4854-9a3a-53798fe56051-kube-api-access-4gfg4\") pod \"s3-init-5n4x5\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") " pod="kserve/s3-init-5n4x5"
Jul 09 17:51:07.883416 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.883162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e083-903b-4854-9a3a-53798fe56051-model-cache\") pod \"s3-init-5n4x5\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") " pod="kserve/s3-init-5n4x5"
Jul 09 17:51:07.983874 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.983819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e083-903b-4854-9a3a-53798fe56051-model-cache\") pod \"s3-init-5n4x5\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") " pod="kserve/s3-init-5n4x5"
Jul 09 17:51:07.983950 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.983885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gfg4\" (UniqueName: \"kubernetes.io/projected/bdc3e083-903b-4854-9a3a-53798fe56051-kube-api-access-4gfg4\") pod \"s3-init-5n4x5\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") " pod="kserve/s3-init-5n4x5"
Jul 09 17:51:07.984231 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.984215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e083-903b-4854-9a3a-53798fe56051-model-cache\") pod \"s3-init-5n4x5\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") " pod="kserve/s3-init-5n4x5"
Jul 09 17:51:07.993889 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:07.993868    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gfg4\" (UniqueName: \"kubernetes.io/projected/bdc3e083-903b-4854-9a3a-53798fe56051-kube-api-access-4gfg4\") pod \"s3-init-5n4x5\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") " pod="kserve/s3-init-5n4x5"
Jul 09 17:51:08.035666 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:08.035646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5n4x5"
Jul 09 17:51:08.159167 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:08.159141    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-5n4x5"]
Jul 09 17:51:08.161448 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:51:08.161427    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdc3e083_903b_4854_9a3a_53798fe56051.slice/crio-b1748136426d0f39f0d03ee6d57c9d347b051a9b294cdfdbe27fe688dcb6dec7 WatchSource:0}: Error finding container b1748136426d0f39f0d03ee6d57c9d347b051a9b294cdfdbe27fe688dcb6dec7: Status 404 returned error can't find the container with id b1748136426d0f39f0d03ee6d57c9d347b051a9b294cdfdbe27fe688dcb6dec7
Jul 09 17:51:08.864982 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:08.864953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5n4x5" event={"ID":"bdc3e083-903b-4854-9a3a-53798fe56051","Type":"ContainerStarted","Data":"b1748136426d0f39f0d03ee6d57c9d347b051a9b294cdfdbe27fe688dcb6dec7"}
Jul 09 17:51:12.886469 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:12.886419    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5n4x5" event={"ID":"bdc3e083-903b-4854-9a3a-53798fe56051","Type":"ContainerStarted","Data":"ca5266a846d6e5ac96da908e1c15c7de2c33a7b3ea553f8c551ea3d480b330e6"}
Jul 09 17:51:13.891104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:13.891075    2577 generic.go:356] "Generic (PLEG): container finished" podID="bdc3e083-903b-4854-9a3a-53798fe56051" containerID="ca5266a846d6e5ac96da908e1c15c7de2c33a7b3ea553f8c551ea3d480b330e6" exitCode=0
Jul 09 17:51:13.891469 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:13.891138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5n4x5" event={"ID":"bdc3e083-903b-4854-9a3a-53798fe56051","Type":"ContainerDied","Data":"ca5266a846d6e5ac96da908e1c15c7de2c33a7b3ea553f8c551ea3d480b330e6"}
Jul 09 17:51:18.916863 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:18.916814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5n4x5" event={"ID":"bdc3e083-903b-4854-9a3a-53798fe56051","Type":"ContainerStarted","Data":"791e0ac3a93474d95b31dec47715b8a449c0abe7c9e59bc9945a50103afa022b"}
Jul 09 17:51:18.936734 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:51:18.936680    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-5n4x5" podStartSLOduration=2.56822249 podStartE2EDuration="11.936665317s" totalImagesPullingTime="8.184853217s" totalInitContainerRuntime="1.18358961s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:51:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:51:18.935131872 +0000 UTC m=+713.125491788" watchObservedRunningTime="2026-07-09 17:51:18.936665317 +0000 UTC m=+713.127025233"
Jul 09 17:52:23.161252 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:23.161222    2577 generic.go:356] "Generic (PLEG): container finished" podID="bdc3e083-903b-4854-9a3a-53798fe56051" containerID="791e0ac3a93474d95b31dec47715b8a449c0abe7c9e59bc9945a50103afa022b" exitCode=0
Jul 09 17:52:23.161675 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:23.161294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5n4x5" event={"ID":"bdc3e083-903b-4854-9a3a-53798fe56051","Type":"ContainerDied","Data":"791e0ac3a93474d95b31dec47715b8a449c0abe7c9e59bc9945a50103afa022b"}
Jul 09 17:52:24.294461 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:24.294435    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5n4x5"
Jul 09 17:52:24.392095 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:24.392073    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gfg4\" (UniqueName: \"kubernetes.io/projected/bdc3e083-903b-4854-9a3a-53798fe56051-kube-api-access-4gfg4\") pod \"bdc3e083-903b-4854-9a3a-53798fe56051\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") "
Jul 09 17:52:24.392238 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:24.392110    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e083-903b-4854-9a3a-53798fe56051-model-cache\") pod \"bdc3e083-903b-4854-9a3a-53798fe56051\" (UID: \"bdc3e083-903b-4854-9a3a-53798fe56051\") "
Jul 09 17:52:24.394263 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:24.394236    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc3e083-903b-4854-9a3a-53798fe56051-kube-api-access-4gfg4" (OuterVolumeSpecName: "kube-api-access-4gfg4") pod "bdc3e083-903b-4854-9a3a-53798fe56051" (UID: "bdc3e083-903b-4854-9a3a-53798fe56051"). InnerVolumeSpecName "kube-api-access-4gfg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:52:24.411015 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:24.410990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdc3e083-903b-4854-9a3a-53798fe56051-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdc3e083-903b-4854-9a3a-53798fe56051" (UID: "bdc3e083-903b-4854-9a3a-53798fe56051"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:24.493118 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:24.493097    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gfg4\" (UniqueName: \"kubernetes.io/projected/bdc3e083-903b-4854-9a3a-53798fe56051-kube-api-access-4gfg4\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:24.493118 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:24.493118    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdc3e083-903b-4854-9a3a-53798fe56051-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:25.170607 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:25.170579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-5n4x5" event={"ID":"bdc3e083-903b-4854-9a3a-53798fe56051","Type":"ContainerDied","Data":"b1748136426d0f39f0d03ee6d57c9d347b051a9b294cdfdbe27fe688dcb6dec7"}
Jul 09 17:52:25.170725 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:25.170620    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1748136426d0f39f0d03ee6d57c9d347b051a9b294cdfdbe27fe688dcb6dec7"
Jul 09 17:52:25.170725 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:25.170586    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-5n4x5"
Jul 09 17:52:38.701204 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.701174    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"]
Jul 09 17:52:38.701561 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.701482    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc3e083-903b-4854-9a3a-53798fe56051" containerName="s3-init"
Jul 09 17:52:38.701561 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.701507    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc3e083-903b-4854-9a3a-53798fe56051" containerName="s3-init"
Jul 09 17:52:38.701561 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.701525    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc3e083-903b-4854-9a3a-53798fe56051" containerName="download-hf-model"
Jul 09 17:52:38.701561 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.701530    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc3e083-903b-4854-9a3a-53798fe56051" containerName="download-hf-model"
Jul 09 17:52:38.701684 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.701573    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc3e083-903b-4854-9a3a-53798fe56051" containerName="s3-init"
Jul 09 17:52:38.704549 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.704532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.708783 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.708764    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 17:52:38.708928 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.708802    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 09 17:52:38.708928 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.708874    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-lk5vb\""
Jul 09 17:52:38.709912 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.709897    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 17:52:38.729379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.726357    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"]
Jul 09 17:52:38.792108 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792212    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqngc\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-kube-api-access-wqngc\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792271 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792562 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.792562 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.792374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.827575 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.827546    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"]
Jul 09 17:52:38.830939 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.830922    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.848410 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.848392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"]
Jul 09 17:52:38.892939 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.892917    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893047 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.892950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893047 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.892972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893161 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893065    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqngc\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-kube-api-access-wqngc\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893161 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893161 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893161 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893357 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893357 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893357 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893513 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893564 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893663 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.893942 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.893922    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.895335 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.895312    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.895419 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.895401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.901756 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.901739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.902120 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.902100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqngc\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-kube-api-access-wqngc\") pod \"router-gateway-1-openshift-default-799f46c59b-bswgn\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:38.994284 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994256    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4zkx\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-kube-api-access-g4zkx\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994380 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994323    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994420 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994420 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994522 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994476    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994557 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994547    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994590 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994566    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:38.994590 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:38.994583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.014284 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.014266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:39.095576 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095546    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.095743 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.095743 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.095743 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.095743 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095694    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.095743 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095720    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096015 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095742    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096015 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095777    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4zkx\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-kube-api-access-g4zkx\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096015 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.095852    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096161 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.096069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096220 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.096192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096343 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.096314    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096687 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.096642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.096894 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.096831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.098880 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.098812    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.098880 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.098871    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.105314 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.105295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.105945 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.105917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4zkx\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-kube-api-access-g4zkx\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rs72c\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.141090 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.141063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:39.156600 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.156570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"]
Jul 09 17:52:39.159235 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:52:39.159214    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f30fd85_056e_4e08_bec7_8cb25ce18c54.slice/crio-a4ad2a3d8c038a9eb7981b6df4deee74d444c033b29d3b78d677dbc52bfcf40d WatchSource:0}: Error finding container a4ad2a3d8c038a9eb7981b6df4deee74d444c033b29d3b78d677dbc52bfcf40d: Status 404 returned error can't find the container with id a4ad2a3d8c038a9eb7981b6df4deee74d444c033b29d3b78d677dbc52bfcf40d
Jul 09 17:52:39.161487 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.161456    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:39.161571 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.161520    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:39.161571 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.161547    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:39.239075 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.230210    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn" event={"ID":"8f30fd85-056e-4e08-bec7-8cb25ce18c54","Type":"ContainerStarted","Data":"a4ad2a3d8c038a9eb7981b6df4deee74d444c033b29d3b78d677dbc52bfcf40d"}
Jul 09 17:52:39.285096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.284665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"]
Jul 09 17:52:39.286812 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:52:39.286767    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4911d2_6c0a_44cb_a64f_c7cfdccaaecf.slice/crio-faf4fc644d14ec5dbb4749a48ee0b8a23c498e380126384460408ffe5b334376 WatchSource:0}: Error finding container faf4fc644d14ec5dbb4749a48ee0b8a23c498e380126384460408ffe5b334376: Status 404 returned error can't find the container with id faf4fc644d14ec5dbb4749a48ee0b8a23c498e380126384460408ffe5b334376
Jul 09 17:52:39.289061 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.289021    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:39.289152 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.289130    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:39.289193 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:39.289179    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:40.236360 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:40.236329    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn" event={"ID":"8f30fd85-056e-4e08-bec7-8cb25ce18c54","Type":"ContainerStarted","Data":"91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962"}
Jul 09 17:52:40.237920 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:40.237896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c" event={"ID":"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf","Type":"ContainerStarted","Data":"dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c"}
Jul 09 17:52:40.238042 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:40.237942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c" event={"ID":"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf","Type":"ContainerStarted","Data":"faf4fc644d14ec5dbb4749a48ee0b8a23c498e380126384460408ffe5b334376"}
Jul 09 17:52:40.262048 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:40.262008    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn" podStartSLOduration=2.26199446 podStartE2EDuration="2.26199446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:52:40.261317293 +0000 UTC m=+794.451677208" watchObservedRunningTime="2026-07-09 17:52:40.26199446 +0000 UTC m=+794.452354375"
Jul 09 17:52:40.285323 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:40.285286    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c" podStartSLOduration=2.285274587 podStartE2EDuration="2.285274587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:52:40.283801532 +0000 UTC m=+794.474161451" watchObservedRunningTime="2026-07-09 17:52:40.285274587 +0000 UTC m=+794.475634502"
Jul 09 17:52:41.014729 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.014696    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:41.019041 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.019016    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:41.142020 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.141993    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:41.146326 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.146306    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:41.242484 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.242465    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:41.242940 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.242913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:41.243086 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.243066    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:41.243307 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.243290    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:41.359931 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:41.359897    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"]
Jul 09 17:52:43.250484 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:43.250446    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn" podUID="8f30fd85-056e-4e08-bec7-8cb25ce18c54" containerName="istio-proxy" containerID="cri-o://91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962" gracePeriod=30
Jul 09 17:52:46.300444 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.300411    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"]
Jul 09 17:52:46.304117 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.304091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.315728 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.315706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"]
Jul 09 17:52:46.456536 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.456741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.456741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3601b8e1-f642-4c31-bbaa-67e494b4de84-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.456741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.456741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.456741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456680    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.456741 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.457105 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.457105 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.456796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67qn\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-kube-api-access-k67qn\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.557298 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.557204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3601b8e1-f642-4c31-bbaa-67e494b4de84-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.558423 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.558403    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.559003 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.558985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.559739 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.559723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.560315 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.560296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.560869 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.560850    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.561350 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.561333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67qn\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-kube-api-access-k67qn\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.561779 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.561759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.562212 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.562195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.563022 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.559663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.563170 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.558330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3601b8e1-f642-4c31-bbaa-67e494b4de84-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.563282 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.562142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.563382 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.558900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.563479 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.561250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.563568 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.562939    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.570804 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.570782    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67qn\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-kube-api-access-k67qn\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.571408 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.571355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.575315 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.575271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-9zb2v\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.614189 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.614162    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:46.868907 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.868702    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"]
Jul 09 17:52:46.887568 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:52:46.887535    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3601b8e1_f642_4c31_bbaa_67e494b4de84.slice/crio-81763474b1bbdf98b73ff3593e2d1591ab71bcf237cf01a1f02184114252b34a WatchSource:0}: Error finding container 81763474b1bbdf98b73ff3593e2d1591ab71bcf237cf01a1f02184114252b34a: Status 404 returned error can't find the container with id 81763474b1bbdf98b73ff3593e2d1591ab71bcf237cf01a1f02184114252b34a
Jul 09 17:52:46.889793 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.889760    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:46.889893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.889829    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:46.889893 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:46.889880    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:47.275994 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:47.275949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" event={"ID":"3601b8e1-f642-4c31-bbaa-67e494b4de84","Type":"ContainerStarted","Data":"6f60ce23822648b803e75e4e9774d66ce67d3f30cb7a4459c6a7faf1375edda7"}
Jul 09 17:52:47.276195 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:47.276023    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" event={"ID":"3601b8e1-f642-4c31-bbaa-67e494b4de84","Type":"ContainerStarted","Data":"81763474b1bbdf98b73ff3593e2d1591ab71bcf237cf01a1f02184114252b34a"}
Jul 09 17:52:47.615314 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:47.615248    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:47.619448 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:47.619428    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:47.641248 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:47.641185    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podStartSLOduration=1.641165747 podStartE2EDuration="1.641165747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:52:47.305296597 +0000 UTC m=+801.495656513" watchObservedRunningTime="2026-07-09 17:52:47.641165747 +0000 UTC m=+801.831525662"
Jul 09 17:52:48.280969 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.280943    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:48.281816 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.281797    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:48.353443 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.353411    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"]
Jul 09 17:52:48.353739 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.353710    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c" podUID="2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" containerName="istio-proxy" containerID="cri-o://dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c" gracePeriod=30
Jul 09 17:52:48.571979 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.571955    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:48.689361 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689330    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqngc\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-kube-api-access-wqngc\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689392    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-podinfo\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-envoy\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689428    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-data\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689447    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-token\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-certs\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689500    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-credential-socket\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-socket\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.689766 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689570    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istiod-ca-cert\") pod \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\" (UID: \"8f30fd85-056e-4e08-bec7-8cb25ce18c54\") "
Jul 09 17:52:48.690187 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689797    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:48.690187 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.689987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:48.690187 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.690060    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:52:48.690187 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.690134    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:48.690389 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.690293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:48.691855 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.691809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:48.691962 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.691879    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-kube-api-access-wqngc" (OuterVolumeSpecName: "kube-api-access-wqngc") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "kube-api-access-wqngc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:52:48.691962 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.691887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 17:52:48.691962 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.691885    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f30fd85-056e-4e08-bec7-8cb25ce18c54" (UID: "8f30fd85-056e-4e08-bec7-8cb25ce18c54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:52:48.790593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790567    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790591    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790599    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790607    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790615    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790623    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790631    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f30fd85-056e-4e08-bec7-8cb25ce18c54-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790639    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f30fd85-056e-4e08-bec7-8cb25ce18c54-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:48.790801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:48.790648    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqngc\" (UniqueName: \"kubernetes.io/projected/8f30fd85-056e-4e08-bec7-8cb25ce18c54-kube-api-access-wqngc\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:49.286586 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.286552    2577 generic.go:356] "Generic (PLEG): container finished" podID="8f30fd85-056e-4e08-bec7-8cb25ce18c54" containerID="91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962" exitCode=0
Jul 09 17:52:49.286757 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.286594    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"
Jul 09 17:52:49.286757 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.286630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn" event={"ID":"8f30fd85-056e-4e08-bec7-8cb25ce18c54","Type":"ContainerDied","Data":"91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962"}
Jul 09 17:52:49.286757 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.286687    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn" event={"ID":"8f30fd85-056e-4e08-bec7-8cb25ce18c54","Type":"ContainerDied","Data":"a4ad2a3d8c038a9eb7981b6df4deee74d444c033b29d3b78d677dbc52bfcf40d"}
Jul 09 17:52:49.286757 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.286716    2577 scope.go:117] "RemoveContainer" containerID="91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962"
Jul 09 17:52:49.295942 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.295925    2577 scope.go:117] "RemoveContainer" containerID="91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962"
Jul 09 17:52:49.296184 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:52:49.296165    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962\": container with ID starting with 91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962 not found: ID does not exist" containerID="91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962"
Jul 09 17:52:49.296240 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.296193    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962"} err="failed to get container status \"91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962\": rpc error: code = NotFound desc = could not find container \"91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962\": container with ID starting with 91e4523b1e146d7bd222d19e1035e8d1f0c3a454ea327197f018978226d07962 not found: ID does not exist"
Jul 09 17:52:49.310497 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.310475    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"]
Jul 09 17:52:49.317709 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:49.317586    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-bswgn"]
Jul 09 17:52:50.417755 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:50.417675    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f30fd85-056e-4e08-bec7-8cb25ce18c54" path="/var/lib/kubelet/pods/8f30fd85-056e-4e08-bec7-8cb25ce18c54/volumes"
Jul 09 17:52:53.737707 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.737406    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:53.849211 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849130    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-data\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849211 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-certs\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849220    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-token\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849256    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-socket\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849295    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4zkx\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-kube-api-access-g4zkx\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istiod-ca-cert\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-envoy\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849390    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-credential-socket\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.849426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.849423    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-podinfo\") pod \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\" (UID: \"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf\") "
Jul 09 17:52:53.851159 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.850134    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:53.851159 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.850293    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-data" (OuterVolumeSpecName: "istio-data") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:53.851159 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.850419    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:53.851159 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.850912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:52:53.853269 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.853222    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:53.855021 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.854981    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-kube-api-access-g4zkx" (OuterVolumeSpecName: "kube-api-access-g4zkx") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "kube-api-access-g4zkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:52:53.857858 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.856920    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 17:52:53.858935 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.858903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:52:53.859019 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.858923    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-token" (OuterVolumeSpecName: "istio-token") pod "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" (UID: "2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.949960    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.949989    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.950004    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.950019    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.950031    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.950045    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4zkx\" (UniqueName: \"kubernetes.io/projected/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-kube-api-access-g4zkx\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.950058    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.950071    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:53.950104 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:53.950083    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:52:54.309884 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.309831    2577 generic.go:356] "Generic (PLEG): container finished" podID="2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" containerID="dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c" exitCode=0
Jul 09 17:52:54.310048 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.309891    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"
Jul 09 17:52:54.310048 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.309873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c" event={"ID":"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf","Type":"ContainerDied","Data":"dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c"}
Jul 09 17:52:54.310048 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.310006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c" event={"ID":"2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf","Type":"ContainerDied","Data":"faf4fc644d14ec5dbb4749a48ee0b8a23c498e380126384460408ffe5b334376"}
Jul 09 17:52:54.310048 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.310029    2577 scope.go:117] "RemoveContainer" containerID="dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c"
Jul 09 17:52:54.318167 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.318148    2577 scope.go:117] "RemoveContainer" containerID="dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c"
Jul 09 17:52:54.318436 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:52:54.318416    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c\": container with ID starting with dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c not found: ID does not exist" containerID="dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c"
Jul 09 17:52:54.318490 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.318446    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c"} err="failed to get container status \"dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c\": rpc error: code = NotFound desc = could not find container \"dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c\": container with ID starting with dd88c4109dd79dde5a1bdcdf7c8bfd965a27a66c9e2b64b7cbec81bdc26fa38c not found: ID does not exist"
Jul 09 17:52:54.333238 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.333210    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"]
Jul 09 17:52:54.340859 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.340817    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rs72c"]
Jul 09 17:52:54.417661 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:54.417636    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" path="/var/lib/kubelet/pods/2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf/volumes"
Jul 09 17:52:58.334368 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:58.334315    2577 generic.go:356] "Generic (PLEG): container finished" podID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerID="6f60ce23822648b803e75e4e9774d66ce67d3f30cb7a4459c6a7faf1375edda7" exitCode=0
Jul 09 17:52:58.334686 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:58.334390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" event={"ID":"3601b8e1-f642-4c31-bbaa-67e494b4de84","Type":"ContainerDied","Data":"6f60ce23822648b803e75e4e9774d66ce67d3f30cb7a4459c6a7faf1375edda7"}
Jul 09 17:52:58.334813 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:58.334797    2577 scope.go:117] "RemoveContainer" containerID="6f60ce23822648b803e75e4e9774d66ce67d3f30cb7a4459c6a7faf1375edda7"
Jul 09 17:52:58.335790 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:58.335753    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:58.335911 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:58.335861    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:58.335911 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:58.335895    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:52:59.355860 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:59.355500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" event={"ID":"3601b8e1-f642-4c31-bbaa-67e494b4de84","Type":"ContainerStarted","Data":"377e3a1d385b298105d9a3838a3e5594dacae47aab50f30679f5d83520eaf902"}
Jul 09 17:52:59.614923 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:59.614822    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:52:59.616059 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:52:59.616023    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Jul 09 17:53:00.615511 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:00.615475    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Jul 09 17:53:01.615155 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:01.615112    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Jul 09 17:53:02.614978 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:02.614931    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Jul 09 17:53:03.281339 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:03.281302    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:53:03.614640 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:03.614561    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Jul 09 17:53:04.618370 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:04.618342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:53:04.619295 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:04.619276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:53:11.328551 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.328521    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"]
Jul 09 17:53:11.328929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.328796    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f30fd85-056e-4e08-bec7-8cb25ce18c54" containerName="istio-proxy"
Jul 09 17:53:11.328929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.328807    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f30fd85-056e-4e08-bec7-8cb25ce18c54" containerName="istio-proxy"
Jul 09 17:53:11.328929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.328824    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" containerName="istio-proxy"
Jul 09 17:53:11.328929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.328830    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" containerName="istio-proxy"
Jul 09 17:53:11.328929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.328895    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f30fd85-056e-4e08-bec7-8cb25ce18c54" containerName="istio-proxy"
Jul 09 17:53:11.328929 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.328905    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4911d2-6c0a-44cb-a64f-c7cfdccaaecf" containerName="istio-proxy"
Jul 09 17:53:11.330922 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.330907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.334960 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.334934    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fg56t\""
Jul 09 17:53:11.334960 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.334951    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 09 17:53:11.335144 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.334977    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:53:11.347207 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.347182    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"]
Jul 09 17:53:11.372799 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.372774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58051e-a744-445d-94b7-ae2164a1b220-tls-certs\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.372936 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.372820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-home\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.372936 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.372879    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-model-cache\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.372936 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.372896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hw9\" (UniqueName: \"kubernetes.io/projected/ca58051e-a744-445d-94b7-ae2164a1b220-kube-api-access-b7hw9\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.372936 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.372921    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-tmp-dir\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.373100 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.372975    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-dshm\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.373100 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.373007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474286 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474424 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474295    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58051e-a744-445d-94b7-ae2164a1b220-tls-certs\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474424 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-home\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474424 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474366    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-model-cache\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474424 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hw9\" (UniqueName: \"kubernetes.io/projected/ca58051e-a744-445d-94b7-ae2164a1b220-kube-api-access-b7hw9\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474628 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-tmp-dir\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474628 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474483    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-dshm\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474733 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474694    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474789 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-home\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474874 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-tmp-dir\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.474936 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.474918    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-model-cache\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.476682 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.476665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-dshm\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.476876 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.476858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58051e-a744-445d-94b7-ae2164a1b220-tls-certs\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.486693 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.486673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hw9\" (UniqueName: \"kubernetes.io/projected/ca58051e-a744-445d-94b7-ae2164a1b220-kube-api-access-b7hw9\") pod \"precise-prefix-cache-test-kserve-56cbd96798-zfl5f\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.641160 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.641100    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:11.679755 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.679726    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"]
Jul 09 17:53:11.682383 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.682361    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.685987 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.685967    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-6lwrg\""
Jul 09 17:53:11.706389 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.706371    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"]
Jul 09 17:53:11.777427 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.777393    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.777596 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.777433    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.777596 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.777453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.777596 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.777574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.777763 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.777644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67bdn\" (UniqueName: \"kubernetes.io/projected/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kube-api-access-67bdn\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.777763 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.777697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.784193 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.784111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"]
Jul 09 17:53:11.786689 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:53:11.786667    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca58051e_a744_445d_94b7_ae2164a1b220.slice/crio-1a77b3741d3e2f83055eb73155a34ca871b24860d45fdae909ce32ffda9eb8cd WatchSource:0}: Error finding container 1a77b3741d3e2f83055eb73155a34ca871b24860d45fdae909ce32ffda9eb8cd: Status 404 returned error can't find the container with id 1a77b3741d3e2f83055eb73155a34ca871b24860d45fdae909ce32ffda9eb8cd
Jul 09 17:53:11.878426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67bdn\" (UniqueName: \"kubernetes.io/projected/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kube-api-access-67bdn\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.878539 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.878539 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.878539 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.878539 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878529    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.878726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.878917 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.879016 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878916    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.879016 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.878946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.879122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.879044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.881105 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.881085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.886748 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.886731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67bdn\" (UniqueName: \"kubernetes.io/projected/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kube-api-access-67bdn\") pod \"precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:11.992023 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:11.991958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:12.124581 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:12.124547    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"]
Jul 09 17:53:12.127254 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:53:12.127216    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16d8f8e1_806d_4314_a3c2_62968cc29d2c.slice/crio-f20995cb4cc17af6c28b492b6f3c008cefcd256e3192ac2cc27bc0bfa4065367 WatchSource:0}: Error finding container f20995cb4cc17af6c28b492b6f3c008cefcd256e3192ac2cc27bc0bfa4065367: Status 404 returned error can't find the container with id f20995cb4cc17af6c28b492b6f3c008cefcd256e3192ac2cc27bc0bfa4065367
Jul 09 17:53:12.417262 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:12.417233    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerStarted","Data":"9b6344f3f4c05841c084445099cbd43238c3e0f9e54a78060a4b8a2c11572894"}
Jul 09 17:53:12.417642 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:12.417278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerStarted","Data":"f20995cb4cc17af6c28b492b6f3c008cefcd256e3192ac2cc27bc0bfa4065367"}
Jul 09 17:53:12.417642 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:12.417294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" event={"ID":"ca58051e-a744-445d-94b7-ae2164a1b220","Type":"ContainerStarted","Data":"1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af"}
Jul 09 17:53:12.417642 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:12.417311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" event={"ID":"ca58051e-a744-445d-94b7-ae2164a1b220","Type":"ContainerStarted","Data":"1a77b3741d3e2f83055eb73155a34ca871b24860d45fdae909ce32ffda9eb8cd"}
Jul 09 17:53:13.419178 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:13.419097    2577 generic.go:356] "Generic (PLEG): container finished" podID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerID="9b6344f3f4c05841c084445099cbd43238c3e0f9e54a78060a4b8a2c11572894" exitCode=0
Jul 09 17:53:13.419516 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:13.419185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerDied","Data":"9b6344f3f4c05841c084445099cbd43238c3e0f9e54a78060a4b8a2c11572894"}
Jul 09 17:53:13.420199 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:13.420182    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:53:15.460638 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:15.460570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerStarted","Data":"46833c08e8f5549beda8651d21886b47e8bd4756c22c3e7a7f3f76ac59e26650"}
Jul 09 17:53:16.468467 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:16.468388    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca58051e-a744-445d-94b7-ae2164a1b220" containerID="1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af" exitCode=0
Jul 09 17:53:16.468944 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:16.468463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" event={"ID":"ca58051e-a744-445d-94b7-ae2164a1b220","Type":"ContainerDied","Data":"1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af"}
Jul 09 17:53:19.490089 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:19.489997    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" event={"ID":"ca58051e-a744-445d-94b7-ae2164a1b220","Type":"ContainerStarted","Data":"0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430"}
Jul 09 17:53:19.520068 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:19.520001    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" podStartSLOduration=1.7799590969999999 podStartE2EDuration="8.519981601s" totalImagesPullingTime="2.521983922s" totalInitContainerRuntime="4.218038582s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:53:19.518956966 +0000 UTC m=+833.709316882" watchObservedRunningTime="2026-07-09 17:53:19.519981601 +0000 UTC m=+833.710341515"
Jul 09 17:53:19.641952 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:19.641906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:21.641643 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:21.641606    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:21.660301 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:21.660272    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:22.506566 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:22.506531    2577 generic.go:356] "Generic (PLEG): container finished" podID="cf762c13-445f-43bd-912e-096388217608" containerID="e189af422966ce7291b2351280e2408ded6482258d3aff7b8c8d725ab6696c13" exitCode=0
Jul 09 17:53:22.506757 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:22.506604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" event={"ID":"cf762c13-445f-43bd-912e-096388217608","Type":"ContainerDied","Data":"e189af422966ce7291b2351280e2408ded6482258d3aff7b8c8d725ab6696c13"}
Jul 09 17:53:22.507312 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:22.507201    2577 scope.go:117] "RemoveContainer" containerID="e189af422966ce7291b2351280e2408ded6482258d3aff7b8c8d725ab6696c13"
Jul 09 17:53:22.508297 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:22.508259    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:53:22.508420 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:22.508348    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:53:22.508420 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:22.508385    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 17:53:22.521098 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:22.521079    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:53:23.515485 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:23.515444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" event={"ID":"cf762c13-445f-43bd-912e-096388217608","Type":"ContainerStarted","Data":"e5225a06bc028f19d374c6d1bebc051a4ef80607c8410bd779d368a207e12019"}
Jul 09 17:53:23.553340 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:23.553051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:53:23.553340 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:23.553173    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-cvzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Jul 09 17:53:23.553340 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:23.553227    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" podUID="cf762c13-445f-43bd-912e-096388217608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Jul 09 17:53:24.551299 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:24.551257    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-cvzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Jul 09 17:53:24.551749 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:24.551319    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" podUID="cf762c13-445f-43bd-912e-096388217608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Jul 09 17:53:25.551731 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:25.551332    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-cvzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Jul 09 17:53:25.551731 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:25.551398    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" podUID="cf762c13-445f-43bd-912e-096388217608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Jul 09 17:53:25.553426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:25.553383    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t_16d8f8e1-806d-4314-a3c2-62968cc29d2c/main/0.log"
Jul 09 17:53:25.553978 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:25.553948    2577 generic.go:356] "Generic (PLEG): container finished" podID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerID="46833c08e8f5549beda8651d21886b47e8bd4756c22c3e7a7f3f76ac59e26650" exitCode=1
Jul 09 17:53:25.554096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:25.554068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerDied","Data":"46833c08e8f5549beda8651d21886b47e8bd4756c22c3e7a7f3f76ac59e26650"}
Jul 09 17:53:26.551886 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:26.551402    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-cvzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Jul 09 17:53:26.551886 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:26.551457    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" podUID="cf762c13-445f-43bd-912e-096388217608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Jul 09 17:53:27.552347 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:27.552298    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-cvzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Jul 09 17:53:27.552808 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:27.552361    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" podUID="cf762c13-445f-43bd-912e-096388217608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Jul 09 17:53:28.195987 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:28.195947    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"]
Jul 09 17:53:28.196286 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:28.196223    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy" containerID="cri-o://377e3a1d385b298105d9a3838a3e5594dacae47aab50f30679f5d83520eaf902" gracePeriod=30
Jul 09 17:53:28.551983 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:28.551945    2577 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-cvzqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused" start-of-body=
Jul 09 17:53:28.552172 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:28.552022    2577 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk" podUID="cf762c13-445f-43bd-912e-096388217608" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.18:15021/healthz/ready\": dial tcp 10.133.0.18:15021: connect: connection refused"
Jul 09 17:53:29.555418 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:29.555385    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:53:29.555852 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:29.555786    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:53:29.556494 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:29.556475    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-cvzqk"
Jul 09 17:53:33.599944 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:33.599909    2577 generic.go:356] "Generic (PLEG): container finished" podID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerID="377e3a1d385b298105d9a3838a3e5594dacae47aab50f30679f5d83520eaf902" exitCode=0
Jul 09 17:53:33.599944 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:33.599948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" event={"ID":"3601b8e1-f642-4c31-bbaa-67e494b4de84","Type":"ContainerDied","Data":"377e3a1d385b298105d9a3838a3e5594dacae47aab50f30679f5d83520eaf902"}
Jul 09 17:53:33.600449 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:33.600036    2577 scope.go:117] "RemoveContainer" containerID="6f60ce23822648b803e75e4e9774d66ce67d3f30cb7a4459c6a7faf1375edda7"
Jul 09 17:53:35.619711 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:35.619649    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 17:53:46.382176 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.382141    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"]
Jul 09 17:53:46.438773 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.438746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"]
Jul 09 17:53:46.438973 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.438892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.441965 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.441893    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 09 17:53:46.442114 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.441965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-l7qgv\""
Jul 09 17:53:46.532949 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.532920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.533117 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.532967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.533117 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.533096    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.533221 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.533133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2hgt\" (UniqueName: \"kubernetes.io/projected/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kube-api-access-q2hgt\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.634160 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.634085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.634313 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.634183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.634313 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.634210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2hgt\" (UniqueName: \"kubernetes.io/projected/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kube-api-access-q2hgt\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.634313 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.634251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.634540 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.634512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.634674 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.634554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.636758 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.636733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.643297 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.643255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2hgt\" (UniqueName: \"kubernetes.io/projected/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kube-api-access-q2hgt\") pod \"auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:46.749422 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:46.749378    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:47.950243 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:47.950221    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:53:48.047392 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047359    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3601b8e1-f642-4c31-bbaa-67e494b4de84-istiod-ca-cert\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.047540 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-certs\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.047540 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k67qn\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-kube-api-access-k67qn\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.047540 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047519    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-socket\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.047746 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047599    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-podinfo\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.047746 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047667    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.047941 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047671    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-credential-socket\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.048045 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.047934    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.048111 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048047    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-data\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.048174 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048013    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3601b8e1-f642-4c31-bbaa-67e494b4de84-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-envoy\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048088    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-token\") pod \"3601b8e1-f642-4c31-bbaa-67e494b4de84\" (UID: \"3601b8e1-f642-4c31-bbaa-67e494b4de84\") "
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048733    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-data" (OuterVolumeSpecName: "istio-data") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048919    2577 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3601b8e1-f642-4c31-bbaa-67e494b4de84-istiod-ca-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048937    2577 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048949    2577 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-workload-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048960    2577 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-credential-socket\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.048993 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.048972    2577 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-data\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.050239 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.050189    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-kube-api-access-k67qn" (OuterVolumeSpecName: "kube-api-access-k67qn") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "kube-api-access-k67qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:53:48.050601 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.050560    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 17:53:48.050700 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.050638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:53:48.051001 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.050983    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-token" (OuterVolumeSpecName: "istio-token") pod "3601b8e1-f642-4c31-bbaa-67e494b4de84" (UID: "3601b8e1-f642-4c31-bbaa-67e494b4de84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:53:48.150249 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.150216    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k67qn\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-kube-api-access-k67qn\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.150379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.150253    2577 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-podinfo\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.150379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.150268    2577 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-envoy\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.150379 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.150282    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3601b8e1-f642-4c31-bbaa-67e494b4de84-istio-token\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:53:48.226446 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.226419    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"]
Jul 09 17:53:48.266239 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:53:48.266211    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27c6e17_3f9c_4e41_a382_7cc953919cb9.slice/crio-9bc42f5aac4b6ea60c0ce7859f3ab0c7bddaf40cce92b3aaf692f7af134a3dee WatchSource:0}: Error finding container 9bc42f5aac4b6ea60c0ce7859f3ab0c7bddaf40cce92b3aaf692f7af134a3dee: Status 404 returned error can't find the container with id 9bc42f5aac4b6ea60c0ce7859f3ab0c7bddaf40cce92b3aaf692f7af134a3dee
Jul 09 17:53:48.681061 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.681020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" event={"ID":"d27c6e17-3f9c-4e41-a382-7cc953919cb9","Type":"ContainerStarted","Data":"c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017"}
Jul 09 17:53:48.681200 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.681096    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" event={"ID":"d27c6e17-3f9c-4e41-a382-7cc953919cb9","Type":"ContainerStarted","Data":"9bc42f5aac4b6ea60c0ce7859f3ab0c7bddaf40cce92b3aaf692f7af134a3dee"}
Jul 09 17:53:48.682827 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.682547    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"
Jul 09 17:53:48.682827 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.682557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v" event={"ID":"3601b8e1-f642-4c31-bbaa-67e494b4de84","Type":"ContainerDied","Data":"81763474b1bbdf98b73ff3593e2d1591ab71bcf237cf01a1f02184114252b34a"}
Jul 09 17:53:48.682827 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.682613    2577 scope.go:117] "RemoveContainer" containerID="377e3a1d385b298105d9a3838a3e5594dacae47aab50f30679f5d83520eaf902"
Jul 09 17:53:48.685253 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.685231    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t_16d8f8e1-806d-4314-a3c2-62968cc29d2c/main/0.log"
Jul 09 17:53:48.685820 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.685799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerStarted","Data":"42db848e75bfcb0c966ce2053bc1367d60c914bad5d2f00ea389d586ddbc57d2"}
Jul 09 17:53:48.686106 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.686095    2577 scope.go:117] "RemoveContainer" containerID="46833c08e8f5549beda8651d21886b47e8bd4756c22c3e7a7f3f76ac59e26650"
Jul 09 17:53:48.751530 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.751506    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"]
Jul 09 17:53:48.758875 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:48.758826    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-9zb2v"]
Jul 09 17:53:49.693821 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:49.693795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t_16d8f8e1-806d-4314-a3c2-62968cc29d2c/main/0.log"
Jul 09 17:53:49.694298 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:49.694264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerStarted","Data":"aee9a3eabec748ccd081082264c3878125a22b751692625e1eca2504e9042c69"}
Jul 09 17:53:49.694518 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:49.694494    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:49.695752 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:49.695729    2577 generic.go:356] "Generic (PLEG): container finished" podID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerID="c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017" exitCode=0
Jul 09 17:53:49.695869 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:49.695813    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" event={"ID":"d27c6e17-3f9c-4e41-a382-7cc953919cb9","Type":"ContainerDied","Data":"c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017"}
Jul 09 17:53:49.726367 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:49.726323    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" podStartSLOduration=2.729555459 podStartE2EDuration="38.726308802s" totalImagesPullingTime="35.075570887s" totalInitContainerRuntime="921.182456ms" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:53:49.7241033 +0000 UTC m=+863.914463227" watchObservedRunningTime="2026-07-09 17:53:49.726308802 +0000 UTC m=+863.916668736"
Jul 09 17:53:50.417393 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:50.417360    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" path="/var/lib/kubelet/pods/3601b8e1-f642-4c31-bbaa-67e494b4de84/volumes"
Jul 09 17:53:50.708941 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:50.708854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" event={"ID":"d27c6e17-3f9c-4e41-a382-7cc953919cb9","Type":"ContainerStarted","Data":"73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2"}
Jul 09 17:53:50.709330 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:50.709186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:53:50.731830 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:50.731695    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" podStartSLOduration=4.079716351 podStartE2EDuration="4.731680731s" totalImagesPullingTime="0s" totalInitContainerRuntime="651.96438ms" isStatefulPod=false podCreationTimestamp="2026-07-09 17:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:53:50.731030559 +0000 UTC m=+864.921390475" watchObservedRunningTime="2026-07-09 17:53:50.731680731 +0000 UTC m=+864.922040647"
Jul 09 17:53:51.992859 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:51.992815    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:53:51.993217 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:53:51.992892    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:54:01.993776 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:54:01.993743    2577 logging.go:55] [core] [Channel #20 SubChannel #21]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.29:9003", ServerName: "10.133.0.29:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.29:9003: connect: connection refused"
Jul 09 17:54:01.995199 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:01.995169    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:54:01.996414 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:01.996383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:54:02.993456 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:02.993421    2577 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.29:9003\" within 1s: context deadline exceeded"
Jul 09 17:54:21.714979 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:21.714898    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:54:22.760245 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:22.760214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:54:24.610027 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.609868    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"]
Jul 09 17:54:24.610483 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.610155    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="tokenizer" containerID="cri-o://42db848e75bfcb0c966ce2053bc1367d60c914bad5d2f00ea389d586ddbc57d2" gracePeriod=60
Jul 09 17:54:24.613401 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.613353    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"]
Jul 09 17:54:24.626826 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:54:24.626780    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 17:54:24.626826 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:54:24.626826 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:54:24.626826 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="aee9a3eabec748ccd081082264c3878125a22b751692625e1eca2504e9042c69" cmd=["/bin/sleep","15"]
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.626862    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" message="" err=<
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:54:24.626879    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main" containerID="cri-o://aee9a3eabec748ccd081082264c3878125a22b751692625e1eca2504e9042c69"
Jul 09 17:54:24.627127 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.626931    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main" containerID="cri-o://aee9a3eabec748ccd081082264c3878125a22b751692625e1eca2504e9042c69" gracePeriod=60
Jul 09 17:54:24.856093 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.856068    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t_16d8f8e1-806d-4314-a3c2-62968cc29d2c/main/0.log"
Jul 09 17:54:24.856395 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.856373    2577 generic.go:356] "Generic (PLEG): container finished" podID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerID="aee9a3eabec748ccd081082264c3878125a22b751692625e1eca2504e9042c69" exitCode=0
Jul 09 17:54:24.856475 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.856414    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerDied","Data":"aee9a3eabec748ccd081082264c3878125a22b751692625e1eca2504e9042c69"}
Jul 09 17:54:24.856512 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:24.856486    2577 scope.go:117] "RemoveContainer" containerID="46833c08e8f5549beda8651d21886b47e8bd4756c22c3e7a7f3f76ac59e26650"
Jul 09 17:54:25.862826 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:25.862798    2577 generic.go:356] "Generic (PLEG): container finished" podID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerID="42db848e75bfcb0c966ce2053bc1367d60c914bad5d2f00ea389d586ddbc57d2" exitCode=0
Jul 09 17:54:25.863173 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:25.862863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerDied","Data":"42db848e75bfcb0c966ce2053bc1367d60c914bad5d2f00ea389d586ddbc57d2"}
Jul 09 17:54:25.960750 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:25.960728    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:54:26.050931 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.050872    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-tmp\") pod \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") "
Jul 09 17:54:26.050931 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.050913    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kserve-provision-location\") pod \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") "
Jul 09 17:54:26.051096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.050939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-cache\") pod \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") "
Jul 09 17:54:26.051096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.050977    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tls-certs\") pod \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") "
Jul 09 17:54:26.051096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.050999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67bdn\" (UniqueName: \"kubernetes.io/projected/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kube-api-access-67bdn\") pod \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") "
Jul 09 17:54:26.051096 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.051021    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-uds\") pod \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\" (UID: \"16d8f8e1-806d-4314-a3c2-62968cc29d2c\") "
Jul 09 17:54:26.051309 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.051261    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "16d8f8e1-806d-4314-a3c2-62968cc29d2c" (UID: "16d8f8e1-806d-4314-a3c2-62968cc29d2c"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:26.051419 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.051385    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "16d8f8e1-806d-4314-a3c2-62968cc29d2c" (UID: "16d8f8e1-806d-4314-a3c2-62968cc29d2c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:26.051514 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.051469    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "16d8f8e1-806d-4314-a3c2-62968cc29d2c" (UID: "16d8f8e1-806d-4314-a3c2-62968cc29d2c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:26.051758 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.051736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "16d8f8e1-806d-4314-a3c2-62968cc29d2c" (UID: "16d8f8e1-806d-4314-a3c2-62968cc29d2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:26.053163 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.053134    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kube-api-access-67bdn" (OuterVolumeSpecName: "kube-api-access-67bdn") pod "16d8f8e1-806d-4314-a3c2-62968cc29d2c" (UID: "16d8f8e1-806d-4314-a3c2-62968cc29d2c"). InnerVolumeSpecName "kube-api-access-67bdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:54:26.053251 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.053218    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16d8f8e1-806d-4314-a3c2-62968cc29d2c" (UID: "16d8f8e1-806d-4314-a3c2-62968cc29d2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:54:26.152001 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.151974    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67bdn\" (UniqueName: \"kubernetes.io/projected/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kube-api-access-67bdn\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:26.152001 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.152000    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:26.152131 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.152009    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-tmp\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:26.152131 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.152018    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:26.152131 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.152026    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tokenizer-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:26.152131 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.152035    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16d8f8e1-806d-4314-a3c2-62968cc29d2c-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:26.248272 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.248251    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:54:26.259979 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.259963    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:54:26.869024 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.868991    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t" event={"ID":"16d8f8e1-806d-4314-a3c2-62968cc29d2c","Type":"ContainerDied","Data":"f20995cb4cc17af6c28b492b6f3c008cefcd256e3192ac2cc27bc0bfa4065367"}
Jul 09 17:54:26.869024 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.869016    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"
Jul 09 17:54:26.869453 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.869076    2577 scope.go:117] "RemoveContainer" containerID="aee9a3eabec748ccd081082264c3878125a22b751692625e1eca2504e9042c69"
Jul 09 17:54:26.877405 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.877385    2577 scope.go:117] "RemoveContainer" containerID="42db848e75bfcb0c966ce2053bc1367d60c914bad5d2f00ea389d586ddbc57d2"
Jul 09 17:54:26.884390 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.884372    2577 scope.go:117] "RemoveContainer" containerID="9b6344f3f4c05841c084445099cbd43238c3e0f9e54a78060a4b8a2c11572894"
Jul 09 17:54:26.888998 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.888979    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"]
Jul 09 17:54:26.892225 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:26.892070    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-64c46df4zp66t"]
Jul 09 17:54:28.417678 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:28.417647    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" path="/var/lib/kubelet/pods/16d8f8e1-806d-4314-a3c2-62968cc29d2c/volumes"
Jul 09 17:54:39.640725 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.640688    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" podUID="ca58051e-a744-445d-94b7-ae2164a1b220" containerName="main" containerID="cri-o://0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430" gracePeriod=45
Jul 09 17:54:39.891589 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.891535    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:54:39.926419 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.926393    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca58051e-a744-445d-94b7-ae2164a1b220" containerID="0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430" exitCode=0
Jul 09 17:54:39.926554 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.926434    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"
Jul 09 17:54:39.926554 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.926438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" event={"ID":"ca58051e-a744-445d-94b7-ae2164a1b220","Type":"ContainerDied","Data":"0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430"}
Jul 09 17:54:39.926677 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.926560    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f" event={"ID":"ca58051e-a744-445d-94b7-ae2164a1b220","Type":"ContainerDied","Data":"1a77b3741d3e2f83055eb73155a34ca871b24860d45fdae909ce32ffda9eb8cd"}
Jul 09 17:54:39.926677 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.926584    2577 scope.go:117] "RemoveContainer" containerID="0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430"
Jul 09 17:54:39.934430 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.934413    2577 scope.go:117] "RemoveContainer" containerID="1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af"
Jul 09 17:54:39.941700 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.941683    2577 scope.go:117] "RemoveContainer" containerID="0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430"
Jul 09 17:54:39.941985 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:54:39.941964    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430\": container with ID starting with 0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430 not found: ID does not exist" containerID="0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430"
Jul 09 17:54:39.942052 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.941993    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430"} err="failed to get container status \"0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430\": rpc error: code = NotFound desc = could not find container \"0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430\": container with ID starting with 0a7f56ebb27f20e7346bc230bd790964caae3d37f38e80e6e1b270914dd0a430 not found: ID does not exist"
Jul 09 17:54:39.942052 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.942011    2577 scope.go:117] "RemoveContainer" containerID="1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af"
Jul 09 17:54:39.942250 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:54:39.942233    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af\": container with ID starting with 1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af not found: ID does not exist" containerID="1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af"
Jul 09 17:54:39.942297 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.942256    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af"} err="failed to get container status \"1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af\": rpc error: code = NotFound desc = could not find container \"1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af\": container with ID starting with 1a58272d3d795508c694fec0dc0f0ab893c33dc4a5a43fd2a0f735ee845352af not found: ID does not exist"
Jul 09 17:54:39.947505 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947488    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-home\") pod \"ca58051e-a744-445d-94b7-ae2164a1b220\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") "
Jul 09 17:54:39.947568 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947514    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-tmp-dir\") pod \"ca58051e-a744-445d-94b7-ae2164a1b220\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") "
Jul 09 17:54:39.947568 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-model-cache\") pod \"ca58051e-a744-445d-94b7-ae2164a1b220\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") "
Jul 09 17:54:39.947568 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947556    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7hw9\" (UniqueName: \"kubernetes.io/projected/ca58051e-a744-445d-94b7-ae2164a1b220-kube-api-access-b7hw9\") pod \"ca58051e-a744-445d-94b7-ae2164a1b220\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") "
Jul 09 17:54:39.947750 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58051e-a744-445d-94b7-ae2164a1b220-tls-certs\") pod \"ca58051e-a744-445d-94b7-ae2164a1b220\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") "
Jul 09 17:54:39.947750 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947606    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-dshm\") pod \"ca58051e-a744-445d-94b7-ae2164a1b220\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") "
Jul 09 17:54:39.947750 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-kserve-provision-location\") pod \"ca58051e-a744-445d-94b7-ae2164a1b220\" (UID: \"ca58051e-a744-445d-94b7-ae2164a1b220\") "
Jul 09 17:54:39.947920 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca58051e-a744-445d-94b7-ae2164a1b220" (UID: "ca58051e-a744-445d-94b7-ae2164a1b220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.947920 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-home" (OuterVolumeSpecName: "home") pod "ca58051e-a744-445d-94b7-ae2164a1b220" (UID: "ca58051e-a744-445d-94b7-ae2164a1b220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.947920 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947802    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca58051e-a744-445d-94b7-ae2164a1b220" (UID: "ca58051e-a744-445d-94b7-ae2164a1b220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.948074 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.947937    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:39.948074 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.948011    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:39.948074 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.948026    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:39.949494 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.949467    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca58051e-a744-445d-94b7-ae2164a1b220-kube-api-access-b7hw9" (OuterVolumeSpecName: "kube-api-access-b7hw9") pod "ca58051e-a744-445d-94b7-ae2164a1b220" (UID: "ca58051e-a744-445d-94b7-ae2164a1b220"). InnerVolumeSpecName "kube-api-access-b7hw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:54:39.949622 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.949602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-dshm" (OuterVolumeSpecName: "dshm") pod "ca58051e-a744-445d-94b7-ae2164a1b220" (UID: "ca58051e-a744-445d-94b7-ae2164a1b220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:39.949717 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:39.949695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca58051e-a744-445d-94b7-ae2164a1b220-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca58051e-a744-445d-94b7-ae2164a1b220" (UID: "ca58051e-a744-445d-94b7-ae2164a1b220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:54:40.002536 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.002513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca58051e-a744-445d-94b7-ae2164a1b220" (UID: "ca58051e-a744-445d-94b7-ae2164a1b220"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:54:40.048828 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.048797    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.048828 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.048827    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hw9\" (UniqueName: \"kubernetes.io/projected/ca58051e-a744-445d-94b7-ae2164a1b220-kube-api-access-b7hw9\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.048949 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.048854    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca58051e-a744-445d-94b7-ae2164a1b220-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.048949 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.048865    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca58051e-a744-445d-94b7-ae2164a1b220-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:54:40.249520 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.249496    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"]
Jul 09 17:54:40.253053 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.253033    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-56cbd96798-zfl5f"]
Jul 09 17:54:40.417539 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:40.417511    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca58051e-a744-445d-94b7-ae2164a1b220" path="/var/lib/kubelet/pods/ca58051e-a744-445d-94b7-ae2164a1b220/volumes"
Jul 09 17:54:50.887042 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887005    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"]
Jul 09 17:54:50.887430 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887413    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main"
Jul 09 17:54:50.887473 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887435    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main"
Jul 09 17:54:50.887515 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887456    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="storage-initializer"
Jul 09 17:54:50.887515 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887484    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="storage-initializer"
Jul 09 17:54:50.887515 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887503    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="tokenizer"
Jul 09 17:54:50.887515 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887512    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="tokenizer"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887532    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca58051e-a744-445d-94b7-ae2164a1b220" containerName="main"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887541    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca58051e-a744-445d-94b7-ae2164a1b220" containerName="main"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887554    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca58051e-a744-445d-94b7-ae2164a1b220" containerName="storage-initializer"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887563    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca58051e-a744-445d-94b7-ae2164a1b220" containerName="storage-initializer"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887574    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887583    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887593    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy"
Jul 09 17:54:50.887650 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887601    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy"
Jul 09 17:54:50.887894 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887662    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy"
Jul 09 17:54:50.887894 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887670    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main"
Jul 09 17:54:50.887894 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887676    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="tokenizer"
Jul 09 17:54:50.887894 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main"
Jul 09 17:54:50.887894 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887688    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3601b8e1-f642-4c31-bbaa-67e494b4de84" containerName="istio-proxy"
Jul 09 17:54:50.887894 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.887693    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca58051e-a744-445d-94b7-ae2164a1b220" containerName="main"
Jul 09 17:54:50.892355 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.892338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:50.894982 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.894962    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 09 17:54:50.899963 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.899940    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"]
Jul 09 17:54:50.921013 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.920987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lktgq\" (UniqueName: \"kubernetes.io/projected/ca003d34-16ae-459c-8e03-7d554eb5a60c-kube-api-access-lktgq\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:50.921117 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.921020    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca003d34-16ae-459c-8e03-7d554eb5a60c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:50.921117 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.921051    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:50.921117 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.921085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:50.921243 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.921148    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:50.921243 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:50.921187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022295 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022435 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022435 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lktgq\" (UniqueName: \"kubernetes.io/projected/ca003d34-16ae-459c-8e03-7d554eb5a60c-kube-api-access-lktgq\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022435 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca003d34-16ae-459c-8e03-7d554eb5a60c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022435 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022708 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022708 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022821 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022802    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.022889 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.022873    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.024611 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.024588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.024751 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.024733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca003d34-16ae-459c-8e03-7d554eb5a60c-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.030289 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.030266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lktgq\" (UniqueName: \"kubernetes.io/projected/ca003d34-16ae-459c-8e03-7d554eb5a60c-kube-api-access-lktgq\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.203616 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.203562    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:54:51.333390 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.333367    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"]
Jul 09 17:54:51.336091 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:54:51.336065    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca003d34_16ae_459c_8e03_7d554eb5a60c.slice/crio-1f3a1b5eafe5cfb0c8d7acfd04d79d1029f25b4c0557867d505cf7a2017951e4 WatchSource:0}: Error finding container 1f3a1b5eafe5cfb0c8d7acfd04d79d1029f25b4c0557867d505cf7a2017951e4: Status 404 returned error can't find the container with id 1f3a1b5eafe5cfb0c8d7acfd04d79d1029f25b4c0557867d505cf7a2017951e4
Jul 09 17:54:51.978103 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.978065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6" event={"ID":"ca003d34-16ae-459c-8e03-7d554eb5a60c","Type":"ContainerStarted","Data":"95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c"}
Jul 09 17:54:51.978471 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:51.978125    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6" event={"ID":"ca003d34-16ae-459c-8e03-7d554eb5a60c","Type":"ContainerStarted","Data":"1f3a1b5eafe5cfb0c8d7acfd04d79d1029f25b4c0557867d505cf7a2017951e4"}
Jul 09 17:54:52.002639 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:52.002591    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6" podStartSLOduration=2.00257498 podStartE2EDuration="2.00257498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:54:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:54:52.001100932 +0000 UTC m=+926.191460849" watchObservedRunningTime="2026-07-09 17:54:52.00257498 +0000 UTC m=+926.192934895"
Jul 09 17:54:52.203952 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:54:52.203920    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:55:01.204369 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:01.204334    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:55:01.216638 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:01.216613    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:55:02.030318 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:02.030292    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:55:26.734044 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:26.733972    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"]
Jul 09 17:55:41.753985 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:41.753918    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6" podUID="ca003d34-16ae-459c-8e03-7d554eb5a60c" containerName="main" containerID="cri-o://95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c" gracePeriod=45
Jul 09 17:55:41.999447 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:41.999422    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:55:42.084498 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084444    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lktgq\" (UniqueName: \"kubernetes.io/projected/ca003d34-16ae-459c-8e03-7d554eb5a60c-kube-api-access-lktgq\") pod \"ca003d34-16ae-459c-8e03-7d554eb5a60c\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") "
Jul 09 17:55:42.084498 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084474    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-home\") pod \"ca003d34-16ae-459c-8e03-7d554eb5a60c\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") "
Jul 09 17:55:42.084669 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084515    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca003d34-16ae-459c-8e03-7d554eb5a60c-tls-certs\") pod \"ca003d34-16ae-459c-8e03-7d554eb5a60c\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") "
Jul 09 17:55:42.084669 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-dshm\") pod \"ca003d34-16ae-459c-8e03-7d554eb5a60c\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") "
Jul 09 17:55:42.084772 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-model-cache\") pod \"ca003d34-16ae-459c-8e03-7d554eb5a60c\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") "
Jul 09 17:55:42.084772 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-tmp-dir\") pod \"ca003d34-16ae-459c-8e03-7d554eb5a60c\" (UID: \"ca003d34-16ae-459c-8e03-7d554eb5a60c\") "
Jul 09 17:55:42.084899 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-home" (OuterVolumeSpecName: "home") pod "ca003d34-16ae-459c-8e03-7d554eb5a60c" (UID: "ca003d34-16ae-459c-8e03-7d554eb5a60c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:55:42.084997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084976    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:55:42.085098 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084982    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ca003d34-16ae-459c-8e03-7d554eb5a60c" (UID: "ca003d34-16ae-459c-8e03-7d554eb5a60c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:55:42.085098 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.084999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ca003d34-16ae-459c-8e03-7d554eb5a60c" (UID: "ca003d34-16ae-459c-8e03-7d554eb5a60c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:55:42.086415 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.086396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca003d34-16ae-459c-8e03-7d554eb5a60c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ca003d34-16ae-459c-8e03-7d554eb5a60c" (UID: "ca003d34-16ae-459c-8e03-7d554eb5a60c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:55:42.086593 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.086575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca003d34-16ae-459c-8e03-7d554eb5a60c-kube-api-access-lktgq" (OuterVolumeSpecName: "kube-api-access-lktgq") pod "ca003d34-16ae-459c-8e03-7d554eb5a60c" (UID: "ca003d34-16ae-459c-8e03-7d554eb5a60c"). InnerVolumeSpecName "kube-api-access-lktgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:55:42.086633 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.086592    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-dshm" (OuterVolumeSpecName: "dshm") pod "ca003d34-16ae-459c-8e03-7d554eb5a60c" (UID: "ca003d34-16ae-459c-8e03-7d554eb5a60c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:55:42.183396 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.183366    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca003d34-16ae-459c-8e03-7d554eb5a60c" containerID="95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c" exitCode=0
Jul 09 17:55:42.183502 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.183415    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"
Jul 09 17:55:42.183502 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.183445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6" event={"ID":"ca003d34-16ae-459c-8e03-7d554eb5a60c","Type":"ContainerDied","Data":"95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c"}
Jul 09 17:55:42.183579 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.183518    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6" event={"ID":"ca003d34-16ae-459c-8e03-7d554eb5a60c","Type":"ContainerDied","Data":"1f3a1b5eafe5cfb0c8d7acfd04d79d1029f25b4c0557867d505cf7a2017951e4"}
Jul 09 17:55:42.183579 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.183539    2577 scope.go:117] "RemoveContainer" containerID="95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c"
Jul 09 17:55:42.185409 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.185394    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lktgq\" (UniqueName: \"kubernetes.io/projected/ca003d34-16ae-459c-8e03-7d554eb5a60c-kube-api-access-lktgq\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:55:42.185467 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.185412    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca003d34-16ae-459c-8e03-7d554eb5a60c-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:55:42.185467 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.185423    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:55:42.185467 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.185431    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:55:42.185467 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.185439    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca003d34-16ae-459c-8e03-7d554eb5a60c-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:55:42.191469 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.191454    2577 scope.go:117] "RemoveContainer" containerID="95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c"
Jul 09 17:55:42.191728 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:55:42.191712    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c\": container with ID starting with 95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c not found: ID does not exist" containerID="95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c"
Jul 09 17:55:42.191772 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.191736    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c"} err="failed to get container status \"95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c\": rpc error: code = NotFound desc = could not find container \"95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c\": container with ID starting with 95d42fc9a4a794b198f588cce0ec5173bccbac97556694eebd9ffd3beda3954c not found: ID does not exist"
Jul 09 17:55:42.204023 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.203998    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"]
Jul 09 17:55:42.207528 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.207506    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-e45d1f79-kserve-7fdbbd4879x5hm6"]
Jul 09 17:55:42.417341 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:55:42.417274    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca003d34-16ae-459c-8e03-7d554eb5a60c" path="/var/lib/kubelet/pods/ca003d34-16ae-459c-8e03-7d554eb5a60c/volumes"
Jul 09 17:56:13.092726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.092691    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"]
Jul 09 17:56:13.093256 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.093015    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca003d34-16ae-459c-8e03-7d554eb5a60c" containerName="main"
Jul 09 17:56:13.093256 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.093027    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca003d34-16ae-459c-8e03-7d554eb5a60c" containerName="main"
Jul 09 17:56:13.093256 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.093043    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main"
Jul 09 17:56:13.093256 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.093049    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16d8f8e1-806d-4314-a3c2-62968cc29d2c" containerName="main"
Jul 09 17:56:13.093256 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.093111    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca003d34-16ae-459c-8e03-7d554eb5a60c" containerName="main"
Jul 09 17:56:13.096032 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.096015    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.098684 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.098664    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 09 17:56:13.107198 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.107176    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"]
Jul 09 17:56:13.195954 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.195930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzdx\" (UniqueName: \"kubernetes.io/projected/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-kube-api-access-2rzdx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.196074 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.195964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.196074 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.195988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.196074 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.196015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.196074 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.196032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.196241 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.196070    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297235 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297417 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzdx\" (UniqueName: \"kubernetes.io/projected/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-kube-api-access-2rzdx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297417 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297417 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297317    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297417 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297417 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297667 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297667 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.297736 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.297714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.299675 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.299652    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.299927 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.299899    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.306215 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.306188    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzdx\" (UniqueName: \"kubernetes.io/projected/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-kube-api-access-2rzdx\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.406999 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.406938    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:13.536569 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:13.536549    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"]
Jul 09 17:56:13.538539 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:56:13.538511    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86daef83_33d0_4a5e_ac54_fc72bd50c4e5.slice/crio-f2d3ea57370fbff464dfd51cb7cb11516400040c400f1f0cae7a79307b6b792c WatchSource:0}: Error finding container f2d3ea57370fbff464dfd51cb7cb11516400040c400f1f0cae7a79307b6b792c: Status 404 returned error can't find the container with id f2d3ea57370fbff464dfd51cb7cb11516400040c400f1f0cae7a79307b6b792c
Jul 09 17:56:14.315124 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:14.315091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg" event={"ID":"86daef83-33d0-4a5e-ac54-fc72bd50c4e5","Type":"ContainerStarted","Data":"43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a"}
Jul 09 17:56:14.315657 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:14.315147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg" event={"ID":"86daef83-33d0-4a5e-ac54-fc72bd50c4e5","Type":"ContainerStarted","Data":"f2d3ea57370fbff464dfd51cb7cb11516400040c400f1f0cae7a79307b6b792c"}
Jul 09 17:56:14.334657 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:14.334615    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg" podStartSLOduration=1.334600206 podStartE2EDuration="1.334600206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:56:14.332946465 +0000 UTC m=+1008.523306380" watchObservedRunningTime="2026-07-09 17:56:14.334600206 +0000 UTC m=+1008.524960122"
Jul 09 17:56:14.408036 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:14.408012    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:23.407994 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:23.407957    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:23.420156 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:23.420129    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:24.369666 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:24.369639    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:56:32.058570 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.058530    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"]
Jul 09 17:56:32.068390 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:56:32.068348    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 17:56:32.068390 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:56:32.068390 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:56:32.068390 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2" cmd=["/bin/sleep","15"]
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.068408    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" message="" err=<
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:56:32.068424    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerName="main" containerID="cri-o://73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2"
Jul 09 17:56:32.068670 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.068500    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerName="main" containerID="cri-o://73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2" gracePeriod=60
Jul 09 17:56:32.308820 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.308764    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:56:32.329381 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.329335    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kserve-provision-location\") pod \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") "
Jul 09 17:56:32.329381 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.329373    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2hgt\" (UniqueName: \"kubernetes.io/projected/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kube-api-access-q2hgt\") pod \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") "
Jul 09 17:56:32.329607 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.329460    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tokenizer-uds\") pod \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") "
Jul 09 17:56:32.329607 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.329520    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tls-certs\") pod \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\" (UID: \"d27c6e17-3f9c-4e41-a382-7cc953919cb9\") "
Jul 09 17:56:32.330060 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.330020    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d27c6e17-3f9c-4e41-a382-7cc953919cb9" (UID: "d27c6e17-3f9c-4e41-a382-7cc953919cb9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:56:32.330489 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.330462    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d27c6e17-3f9c-4e41-a382-7cc953919cb9" (UID: "d27c6e17-3f9c-4e41-a382-7cc953919cb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:56:32.332269 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.332247    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kube-api-access-q2hgt" (OuterVolumeSpecName: "kube-api-access-q2hgt") pod "d27c6e17-3f9c-4e41-a382-7cc953919cb9" (UID: "d27c6e17-3f9c-4e41-a382-7cc953919cb9"). InnerVolumeSpecName "kube-api-access-q2hgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:56:32.332784 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.332748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d27c6e17-3f9c-4e41-a382-7cc953919cb9" (UID: "d27c6e17-3f9c-4e41-a382-7cc953919cb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:56:32.392036 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.392005    2577 generic.go:356] "Generic (PLEG): container finished" podID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerID="73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2" exitCode=0
Jul 09 17:56:32.392167 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.392060    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"
Jul 09 17:56:32.392167 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.392079    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" event={"ID":"d27c6e17-3f9c-4e41-a382-7cc953919cb9","Type":"ContainerDied","Data":"73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2"}
Jul 09 17:56:32.392167 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.392138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht" event={"ID":"d27c6e17-3f9c-4e41-a382-7cc953919cb9","Type":"ContainerDied","Data":"9bc42f5aac4b6ea60c0ce7859f3ab0c7bddaf40cce92b3aaf692f7af134a3dee"}
Jul 09 17:56:32.392167 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.392162    2577 scope.go:117] "RemoveContainer" containerID="73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2"
Jul 09 17:56:32.400423 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.400403    2577 scope.go:117] "RemoveContainer" containerID="c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017"
Jul 09 17:56:32.407392 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.407378    2577 scope.go:117] "RemoveContainer" containerID="73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2"
Jul 09 17:56:32.407640 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:56:32.407622    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2\": container with ID starting with 73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2 not found: ID does not exist" containerID="73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2"
Jul 09 17:56:32.407684 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.407646    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2"} err="failed to get container status \"73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2\": rpc error: code = NotFound desc = could not find container \"73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2\": container with ID starting with 73d7a62789f6e562e10f56c59e23e5e05ac0a6e27cdfe6313f8f68a4613d98f2 not found: ID does not exist"
Jul 09 17:56:32.407684 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.407663    2577 scope.go:117] "RemoveContainer" containerID="c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017"
Jul 09 17:56:32.407931 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:56:32.407913    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017\": container with ID starting with c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017 not found: ID does not exist" containerID="c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017"
Jul 09 17:56:32.407997 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.407938    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017"} err="failed to get container status \"c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017\": rpc error: code = NotFound desc = could not find container \"c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017\": container with ID starting with c223bbe6312a99c44f549ac5a9009d81e0ac9819a32936e438793c09b1297017 not found: ID does not exist"
Jul 09 17:56:32.417485 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.417468    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"]
Jul 09 17:56:32.418097 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.418081    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-6f9bd5659b-zpjht"]
Jul 09 17:56:32.430621 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.430593    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:56:32.430621 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.430612    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d27c6e17-3f9c-4e41-a382-7cc953919cb9-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:56:32.430621 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.430620    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:56:32.430818 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:32.430630    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2hgt\" (UniqueName: \"kubernetes.io/projected/d27c6e17-3f9c-4e41-a382-7cc953919cb9-kube-api-access-q2hgt\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:56:34.417019 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:34.416989    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" path="/var/lib/kubelet/pods/d27c6e17-3f9c-4e41-a382-7cc953919cb9/volumes"
Jul 09 17:56:55.600659 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.600626    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"]
Jul 09 17:56:55.601223 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.600901    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerName="main"
Jul 09 17:56:55.601223 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.600911    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerName="main"
Jul 09 17:56:55.601223 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.600927    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerName="storage-initializer"
Jul 09 17:56:55.601223 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.600934    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerName="storage-initializer"
Jul 09 17:56:55.601223 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.600986    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d27c6e17-3f9c-4e41-a382-7cc953919cb9" containerName="main"
Jul 09 17:56:55.603630 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.603607    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.607354 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.607329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-xkrgn\""
Jul 09 17:56:55.607457 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.607368    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 09 17:56:55.613824 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.613802    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"]
Jul 09 17:56:55.699851 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.699812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.699976 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.699950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bnsj\" (UniqueName: \"kubernetes.io/projected/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-kube-api-access-4bnsj\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.700056 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.700039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.801065 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.801039    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.801163 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.801069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.801163 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.801104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bnsj\" (UniqueName: \"kubernetes.io/projected/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-kube-api-access-4bnsj\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.801400 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.801378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.803432 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.803409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.809179 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.809147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bnsj\" (UniqueName: \"kubernetes.io/projected/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-kube-api-access-4bnsj\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:55.912345 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:55.912292    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:56.039906 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:56.039881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"]
Jul 09 17:56:56.042247 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:56:56.042206    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4736c7_ccf9_4b03_b6b8_aeb830cfee6e.slice/crio-9d9094b5f79c4629907ee4bf59b030d76a5a52b80667e1c7731bd06e2541178c WatchSource:0}: Error finding container 9d9094b5f79c4629907ee4bf59b030d76a5a52b80667e1c7731bd06e2541178c: Status 404 returned error can't find the container with id 9d9094b5f79c4629907ee4bf59b030d76a5a52b80667e1c7731bd06e2541178c
Jul 09 17:56:56.489937 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:56.489896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" event={"ID":"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e","Type":"ContainerStarted","Data":"c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e"}
Jul 09 17:56:56.490153 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:56.489951    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" event={"ID":"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e","Type":"ContainerStarted","Data":"9d9094b5f79c4629907ee4bf59b030d76a5a52b80667e1c7731bd06e2541178c"}
Jul 09 17:56:56.490153 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:56.489997    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:56:56.508289 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:56:56.508242    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" podStartSLOduration=1.508226964 podStartE2EDuration="1.508226964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:56:56.506619507 +0000 UTC m=+1050.696979423" watchObservedRunningTime="2026-07-09 17:56:56.508226964 +0000 UTC m=+1050.698586879"
Jul 09 17:57:20.217085 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.217010    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"]
Jul 09 17:57:20.220664 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.220641    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.223746 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.223722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-nt7bp\""
Jul 09 17:57:20.223876 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.223725    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:57:20.223876 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.223780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 09 17:57:20.231796 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.231765    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"]
Jul 09 17:57:20.371326 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.371297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.371459 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.371331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2z7\" (UniqueName: \"kubernetes.io/projected/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kube-api-access-ml2z7\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.371459 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.371368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.371459 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.371410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.472426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.472363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.472426 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.472408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.472612 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.472426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2z7\" (UniqueName: \"kubernetes.io/projected/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kube-api-access-ml2z7\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.472612 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.472463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.472752 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.472731    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.472812 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.472755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.474823 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.474800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.481137 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.481110    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2z7\" (UniqueName: \"kubernetes.io/projected/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kube-api-access-ml2z7\") pod \"auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.531938 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.531916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:20.663414 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:20.663387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"]
Jul 09 17:57:20.665158 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:57:20.665133    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a6194e_29cc_44d9_b63f_dd6560cc6bdc.slice/crio-a5b576bad2b6c160d3eaac9b2fc5b2cfbb407f29e54f280c425cfeb27a6ece5c WatchSource:0}: Error finding container a5b576bad2b6c160d3eaac9b2fc5b2cfbb407f29e54f280c425cfeb27a6ece5c: Status 404 returned error can't find the container with id a5b576bad2b6c160d3eaac9b2fc5b2cfbb407f29e54f280c425cfeb27a6ece5c
Jul 09 17:57:21.588391 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:21.588317    2577 generic.go:356] "Generic (PLEG): container finished" podID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerID="96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571" exitCode=0
Jul 09 17:57:21.588391 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:21.588361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" event={"ID":"35a6194e-29cc-44d9-b63f-dd6560cc6bdc","Type":"ContainerDied","Data":"96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571"}
Jul 09 17:57:21.588785 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:21.588396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" event={"ID":"35a6194e-29cc-44d9-b63f-dd6560cc6bdc","Type":"ContainerStarted","Data":"a5b576bad2b6c160d3eaac9b2fc5b2cfbb407f29e54f280c425cfeb27a6ece5c"}
Jul 09 17:57:22.594579 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:22.594547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" event={"ID":"35a6194e-29cc-44d9-b63f-dd6560cc6bdc","Type":"ContainerStarted","Data":"1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f"}
Jul 09 17:57:22.594974 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:22.594755    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:57:22.620985 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:22.620940    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" podStartSLOduration=2.004310173 podStartE2EDuration="2.620924938s" totalImagesPullingTime="0s" totalInitContainerRuntime="616.614765ms" isStatefulPod=false podCreationTimestamp="2026-07-09 17:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:57:22.618674224 +0000 UTC m=+1076.809034138" watchObservedRunningTime="2026-07-09 17:57:22.620924938 +0000 UTC m=+1076.811284852"
Jul 09 17:57:27.495311 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:27.495285    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:57:42.015687 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.015645    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"]
Jul 09 17:57:42.024700 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:57:42.024667    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 17:57:42.024700 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:57:42.024700 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:57:42.024700 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e" cmd=["/bin/sleep","15"]
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.024714    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" message="" err=<
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:57:42.024726    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" podUID="ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" containerName="main" containerID="cri-o://c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e"
Jul 09 17:57:42.024946 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.024775    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" podUID="ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" containerName="main" containerID="cri-o://c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e" gracePeriod=60
Jul 09 17:57:42.271832 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.271772    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:57:42.334573 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.334547    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tls-certs\") pod \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") "
Jul 09 17:57:42.334703 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.334577    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tokenizer-uds\") pod \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") "
Jul 09 17:57:42.334703 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.334612    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bnsj\" (UniqueName: \"kubernetes.io/projected/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-kube-api-access-4bnsj\") pod \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\" (UID: \"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e\") "
Jul 09 17:57:42.334823 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.334790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" (UID: "ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:57:42.336554 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.336530    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-kube-api-access-4bnsj" (OuterVolumeSpecName: "kube-api-access-4bnsj") pod "ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" (UID: "ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e"). InnerVolumeSpecName "kube-api-access-4bnsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:57:42.336626 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.336573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" (UID: "ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:57:42.435348 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.435327    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:57:42.435348 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.435346    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:57:42.435478 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.435355    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bnsj\" (UniqueName: \"kubernetes.io/projected/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e-kube-api-access-4bnsj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:57:42.680801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.680734    2577 generic.go:356] "Generic (PLEG): container finished" podID="ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" containerID="c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e" exitCode=0
Jul 09 17:57:42.680801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.680770    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" event={"ID":"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e","Type":"ContainerDied","Data":"c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e"}
Jul 09 17:57:42.680801 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.680777    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"
Jul 09 17:57:42.681014 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.680808    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7" event={"ID":"ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e","Type":"ContainerDied","Data":"9d9094b5f79c4629907ee4bf59b030d76a5a52b80667e1c7731bd06e2541178c"}
Jul 09 17:57:42.681014 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.680828    2577 scope.go:117] "RemoveContainer" containerID="c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e"
Jul 09 17:57:42.689033 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.689016    2577 scope.go:117] "RemoveContainer" containerID="c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e"
Jul 09 17:57:42.689270 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:57:42.689250    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e\": container with ID starting with c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e not found: ID does not exist" containerID="c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e"
Jul 09 17:57:42.689337 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.689282    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e"} err="failed to get container status \"c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e\": rpc error: code = NotFound desc = could not find container \"c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e\": container with ID starting with c0e9efeed949857d85d1f7f3c5476e8768d71358f8313c1e4991232eea731f3e not found: ID does not exist"
Jul 09 17:57:42.697116 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.697095    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"]
Jul 09 17:57:42.699180 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:42.699162    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheztzw7"]
Jul 09 17:57:44.417192 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:44.417157    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" path="/var/lib/kubelet/pods/ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e/volumes"
Jul 09 17:57:53.601041 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:57:53.601007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:58:00.913358 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:00.913327    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"]
Jul 09 17:58:15.933333 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:15.933293    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg" podUID="86daef83-33d0-4a5e-ac54-fc72bd50c4e5" containerName="main" containerID="cri-o://43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a" gracePeriod=45
Jul 09 17:58:16.179346 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.179324    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:58:16.254739 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.254710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tls-certs\") pod \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") "
Jul 09 17:58:16.254885 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.254753    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rzdx\" (UniqueName: \"kubernetes.io/projected/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-kube-api-access-2rzdx\") pod \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") "
Jul 09 17:58:16.254885 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.254772    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tmp-dir\") pod \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") "
Jul 09 17:58:16.254885 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.254824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-dshm\") pod \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") "
Jul 09 17:58:16.254885 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.254863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-home\") pod \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") "
Jul 09 17:58:16.255122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.254919    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-model-cache\") pod \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\" (UID: \"86daef83-33d0-4a5e-ac54-fc72bd50c4e5\") "
Jul 09 17:58:16.255122 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.255085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86daef83-33d0-4a5e-ac54-fc72bd50c4e5" (UID: "86daef83-33d0-4a5e-ac54-fc72bd50c4e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:58:16.255232 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.255193    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-home" (OuterVolumeSpecName: "home") pod "86daef83-33d0-4a5e-ac54-fc72bd50c4e5" (UID: "86daef83-33d0-4a5e-ac54-fc72bd50c4e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:58:16.255301 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.255270    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "86daef83-33d0-4a5e-ac54-fc72bd50c4e5" (UID: "86daef83-33d0-4a5e-ac54-fc72bd50c4e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:58:16.256896 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.256864    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-kube-api-access-2rzdx" (OuterVolumeSpecName: "kube-api-access-2rzdx") pod "86daef83-33d0-4a5e-ac54-fc72bd50c4e5" (UID: "86daef83-33d0-4a5e-ac54-fc72bd50c4e5"). InnerVolumeSpecName "kube-api-access-2rzdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:58:16.256981 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.256865    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-dshm" (OuterVolumeSpecName: "dshm") pod "86daef83-33d0-4a5e-ac54-fc72bd50c4e5" (UID: "86daef83-33d0-4a5e-ac54-fc72bd50c4e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:58:16.256981 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.256942    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86daef83-33d0-4a5e-ac54-fc72bd50c4e5" (UID: "86daef83-33d0-4a5e-ac54-fc72bd50c4e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:58:16.355789 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.355770    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:58:16.355789 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.355788    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:58:16.355930 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.355796    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:58:16.355930 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.355806    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:58:16.355930 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.355814    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzdx\" (UniqueName: \"kubernetes.io/projected/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-kube-api-access-2rzdx\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:58:16.355930 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.355856    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86daef83-33d0-4a5e-ac54-fc72bd50c4e5-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:58:16.822535 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.822509    2577 generic.go:356] "Generic (PLEG): container finished" podID="86daef83-33d0-4a5e-ac54-fc72bd50c4e5" containerID="43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a" exitCode=0
Jul 09 17:58:16.822651 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.822549    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"
Jul 09 17:58:16.822651 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.822604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg" event={"ID":"86daef83-33d0-4a5e-ac54-fc72bd50c4e5","Type":"ContainerDied","Data":"43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a"}
Jul 09 17:58:16.822765 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.822661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg" event={"ID":"86daef83-33d0-4a5e-ac54-fc72bd50c4e5","Type":"ContainerDied","Data":"f2d3ea57370fbff464dfd51cb7cb11516400040c400f1f0cae7a79307b6b792c"}
Jul 09 17:58:16.822765 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.822694    2577 scope.go:117] "RemoveContainer" containerID="43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a"
Jul 09 17:58:16.830725 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.830711    2577 scope.go:117] "RemoveContainer" containerID="43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a"
Jul 09 17:58:16.830991 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:58:16.830967    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a\": container with ID starting with 43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a not found: ID does not exist" containerID="43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a"
Jul 09 17:58:16.831057 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.830996    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a"} err="failed to get container status \"43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a\": rpc error: code = NotFound desc = could not find container \"43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a\": container with ID starting with 43e174440a932c79f457dbc8778aaf652fefb2f24285c1e830c6b8d686eaf68a not found: ID does not exist"
Jul 09 17:58:16.842243 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.842221    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"]
Jul 09 17:58:16.848273 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:16.848252    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-5b1e8f15-kserve-64df7bddb4z99mg"]
Jul 09 17:58:18.416717 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:18.416680    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86daef83-33d0-4a5e-ac54-fc72bd50c4e5" path="/var/lib/kubelet/pods/86daef83-33d0-4a5e-ac54-fc72bd50c4e5/volumes"
Jul 09 17:58:25.701588 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.701548    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"]
Jul 09 17:58:25.704009 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.701973    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86daef83-33d0-4a5e-ac54-fc72bd50c4e5" containerName="main"
Jul 09 17:58:25.704009 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.701992    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="86daef83-33d0-4a5e-ac54-fc72bd50c4e5" containerName="main"
Jul 09 17:58:25.704009 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.702001    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" containerName="main"
Jul 09 17:58:25.704009 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.702007    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" containerName="main"
Jul 09 17:58:25.704009 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.702088    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad4736c7-ccf9-4b03-b6b8-aeb830cfee6e" containerName="main"
Jul 09 17:58:25.704009 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.702096    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="86daef83-33d0-4a5e-ac54-fc72bd50c4e5" containerName="main"
Jul 09 17:58:25.704852 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.704822    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.708051 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.708031    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 09 17:58:25.716351 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.716330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"]
Jul 09 17:58:25.817344 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.817321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.817443 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.817352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.817443 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.817377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bccce0d-2b49-4d99-934b-ae0d7a946379-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.817518 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.817450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.817518 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.817499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4psjh\" (UniqueName: \"kubernetes.io/projected/5bccce0d-2b49-4d99-934b-ae0d7a946379-kube-api-access-4psjh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.817586 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.817516    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.918628 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.918600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bccce0d-2b49-4d99-934b-ae0d7a946379-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.918726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.918632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.918726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.918664    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4psjh\" (UniqueName: \"kubernetes.io/projected/5bccce0d-2b49-4d99-934b-ae0d7a946379-kube-api-access-4psjh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.918726 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.918686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.918887 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.918733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.918887 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.918763    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.919078 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.919054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.919175 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.919148    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.919230 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.919186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.920948 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.920925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.921089 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.921065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bccce0d-2b49-4d99-934b-ae0d7a946379-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:25.927024 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:25.926983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4psjh\" (UniqueName: \"kubernetes.io/projected/5bccce0d-2b49-4d99-934b-ae0d7a946379-kube-api-access-4psjh\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") " pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:26.014548 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:26.014523    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:26.143233 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:26.143211    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"]
Jul 09 17:58:26.145781 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:58:26.145750    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bccce0d_2b49_4d99_934b_ae0d7a946379.slice/crio-8c59811df57b71b1660933714940e2059975ce5a666d6af6339ecf44ba01270b WatchSource:0}: Error finding container 8c59811df57b71b1660933714940e2059975ce5a666d6af6339ecf44ba01270b: Status 404 returned error can't find the container with id 8c59811df57b71b1660933714940e2059975ce5a666d6af6339ecf44ba01270b
Jul 09 17:58:26.866051 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:26.866020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv" event={"ID":"5bccce0d-2b49-4d99-934b-ae0d7a946379","Type":"ContainerStarted","Data":"b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac"}
Jul 09 17:58:26.866423 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:26.866072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv" event={"ID":"5bccce0d-2b49-4d99-934b-ae0d7a946379","Type":"ContainerStarted","Data":"8c59811df57b71b1660933714940e2059975ce5a666d6af6339ecf44ba01270b"}
Jul 09 17:58:26.889078 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:26.889034    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv" podStartSLOduration=1.8890204160000001 podStartE2EDuration="1.889020416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:58:26.886916701 +0000 UTC m=+1141.077276615" watchObservedRunningTime="2026-07-09 17:58:26.889020416 +0000 UTC m=+1141.079380331"
Jul 09 17:58:27.014735 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:27.014709    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:36.015335 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:36.015303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:36.027717 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:36.027688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:58:36.924968 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:58:36.924940    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 17:59:21.556432 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.556398    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"]
Jul 09 17:59:21.566199 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:59:21.566167    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 17:59:21.566199 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:59:21.566199 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:59:21.566199 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f" cmd=["/bin/sleep","15"]
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.566213    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" message="" err=<
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:59:21.566227    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerName="main" containerID="cri-o://1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f"
Jul 09 17:59:21.566447 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.566264    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerName="main" containerID="cri-o://1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f" gracePeriod=60
Jul 09 17:59:21.811790 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.811732    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:59:21.898612 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.898587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tls-certs\") pod \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") "
Jul 09 17:59:21.898721 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.898628    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tokenizer-uds\") pod \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") "
Jul 09 17:59:21.898721 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.898654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kserve-provision-location\") pod \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") "
Jul 09 17:59:21.898892 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.898872    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml2z7\" (UniqueName: \"kubernetes.io/projected/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kube-api-access-ml2z7\") pod \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\" (UID: \"35a6194e-29cc-44d9-b63f-dd6560cc6bdc\") "
Jul 09 17:59:21.898949 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.898931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "35a6194e-29cc-44d9-b63f-dd6560cc6bdc" (UID: "35a6194e-29cc-44d9-b63f-dd6560cc6bdc"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:59:21.899088 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.899074    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:59:21.899358 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.899340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "35a6194e-29cc-44d9-b63f-dd6560cc6bdc" (UID: "35a6194e-29cc-44d9-b63f-dd6560cc6bdc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:59:21.900682 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.900656    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35a6194e-29cc-44d9-b63f-dd6560cc6bdc" (UID: "35a6194e-29cc-44d9-b63f-dd6560cc6bdc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:59:21.900769 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.900722    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kube-api-access-ml2z7" (OuterVolumeSpecName: "kube-api-access-ml2z7") pod "35a6194e-29cc-44d9-b63f-dd6560cc6bdc" (UID: "35a6194e-29cc-44d9-b63f-dd6560cc6bdc"). InnerVolumeSpecName "kube-api-access-ml2z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:59:21.999740 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.999713    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:59:21.999740 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.999738    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:59:21.999907 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:21.999753    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml2z7\" (UniqueName: \"kubernetes.io/projected/35a6194e-29cc-44d9-b63f-dd6560cc6bdc-kube-api-access-ml2z7\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 17:59:22.105762 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.105691    2577 generic.go:356] "Generic (PLEG): container finished" podID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerID="1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f" exitCode=0
Jul 09 17:59:22.105883 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.105759    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"
Jul 09 17:59:22.105883 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.105761    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" event={"ID":"35a6194e-29cc-44d9-b63f-dd6560cc6bdc","Type":"ContainerDied","Data":"1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f"}
Jul 09 17:59:22.105995 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.105885    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4" event={"ID":"35a6194e-29cc-44d9-b63f-dd6560cc6bdc","Type":"ContainerDied","Data":"a5b576bad2b6c160d3eaac9b2fc5b2cfbb407f29e54f280c425cfeb27a6ece5c"}
Jul 09 17:59:22.105995 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.105906    2577 scope.go:117] "RemoveContainer" containerID="1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f"
Jul 09 17:59:22.115187 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.115173    2577 scope.go:117] "RemoveContainer" containerID="96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571"
Jul 09 17:59:22.122548 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.122532    2577 scope.go:117] "RemoveContainer" containerID="1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f"
Jul 09 17:59:22.122780 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:59:22.122766    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f\": container with ID starting with 1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f not found: ID does not exist" containerID="1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f"
Jul 09 17:59:22.122819 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.122787    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f"} err="failed to get container status \"1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f\": rpc error: code = NotFound desc = could not find container \"1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f\": container with ID starting with 1423526312dbc2d10598b23c4e315c8c876a50771ef05f70e1f729d312f4810f not found: ID does not exist"
Jul 09 17:59:22.122819 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.122801    2577 scope.go:117] "RemoveContainer" containerID="96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571"
Jul 09 17:59:22.123039 ip-10-0-141-44 kubenswrapper[2577]: E0709 17:59:22.123022    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571\": container with ID starting with 96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571 not found: ID does not exist" containerID="96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571"
Jul 09 17:59:22.123077 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.123046    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571"} err="failed to get container status \"96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571\": rpc error: code = NotFound desc = could not find container \"96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571\": container with ID starting with 96885eb181da00bba00ec60861a90c0804331a7862f90f28426cffb453239571 not found: ID does not exist"
Jul 09 17:59:22.133901 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.133882    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"]
Jul 09 17:59:22.138588 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.138570    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-6b9cbc8d98dtpp4"]
Jul 09 17:59:22.420693 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:22.420600    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" path="/var/lib/kubelet/pods/35a6194e-29cc-44d9-b63f-dd6560cc6bdc/volumes"
Jul 09 17:59:26.273930 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:26.273904    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:59:26.285130 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:26.285106    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 17:59:52.725659 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.725628    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"]
Jul 09 17:59:52.726230 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.726022    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerName="main"
Jul 09 17:59:52.726230 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.726041    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerName="main"
Jul 09 17:59:52.726230 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.726070    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerName="storage-initializer"
Jul 09 17:59:52.726230 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.726078    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerName="storage-initializer"
Jul 09 17:59:52.726230 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.726159    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="35a6194e-29cc-44d9-b63f-dd6560cc6bdc" containerName="main"
Jul 09 17:59:52.730807 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.730786    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.733528 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.733508    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 17:59:52.733776 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.733752    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-crbj9\""
Jul 09 17:59:52.733870 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.733825    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 09 17:59:52.739730 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.739707    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"]
Jul 09 17:59:52.814278 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.814218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.814399 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.814281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.814399 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.814313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6v8h\" (UniqueName: \"kubernetes.io/projected/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kube-api-access-m6v8h\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.814399 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.814352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.915081 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.915055    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.915212 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.915093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6v8h\" (UniqueName: \"kubernetes.io/projected/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kube-api-access-m6v8h\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.915212 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.915133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.915212 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.915186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.915559 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.915536    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.915625 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.915574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.917652 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.917624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:52.925558 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:52.925538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6v8h\" (UniqueName: \"kubernetes.io/projected/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kube-api-access-m6v8h\") pod \"auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:53.041531 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:53.041510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:53.179011 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:53.178984    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"]
Jul 09 17:59:53.181339 ip-10-0-141-44 kubenswrapper[2577]: W0709 17:59:53.181315    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f8ddcd_132c_4e06_b5d3_3f34d2e275a4.slice/crio-63049cecb2c3b4994441c9cf33fb7f07852b17a8ec663b02817dab04434f07e9 WatchSource:0}: Error finding container 63049cecb2c3b4994441c9cf33fb7f07852b17a8ec663b02817dab04434f07e9: Status 404 returned error can't find the container with id 63049cecb2c3b4994441c9cf33fb7f07852b17a8ec663b02817dab04434f07e9
Jul 09 17:59:53.235150 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:53.235119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" event={"ID":"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4","Type":"ContainerStarted","Data":"63049cecb2c3b4994441c9cf33fb7f07852b17a8ec663b02817dab04434f07e9"}
Jul 09 17:59:54.241023 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:54.240934    2577 generic.go:356] "Generic (PLEG): container finished" podID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerID="41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb" exitCode=0
Jul 09 17:59:54.241350 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:54.241025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" event={"ID":"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4","Type":"ContainerDied","Data":"41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb"}
Jul 09 17:59:55.246827 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:55.246790    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" event={"ID":"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4","Type":"ContainerStarted","Data":"fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c"}
Jul 09 17:59:55.247201 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:55.246878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 17:59:55.268744 ip-10-0-141-44 kubenswrapper[2577]: I0709 17:59:55.268697    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" podStartSLOduration=2.6174367370000002 podStartE2EDuration="3.268684349s" totalImagesPullingTime="0s" totalInitContainerRuntime="651.247612ms" isStatefulPod=false podCreationTimestamp="2026-07-09 17:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:59:55.267904032 +0000 UTC m=+1229.458263968" watchObservedRunningTime="2026-07-09 17:59:55.268684349 +0000 UTC m=+1229.459044263"
Jul 09 18:00:07.290398 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.290364    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"]
Jul 09 18:00:07.293807 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.293783    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.296634 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.296613    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 09 18:00:07.304007 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.303983    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"]
Jul 09 18:00:07.417572 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.417543    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.417704 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.417575    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.417704 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.417598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjs44\" (UniqueName: \"kubernetes.io/projected/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-kube-api-access-bjs44\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.417704 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.417697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.417894 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.417721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.417894 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.417792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518149 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518121    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjs44\" (UniqueName: \"kubernetes.io/projected/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-kube-api-access-bjs44\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518273 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518273 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518273 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518273 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518254    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518492 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518279    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518600 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518668 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.518668 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.518642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.520562 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.520534    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.520654 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.520641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.527360 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.527340    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjs44\" (UniqueName: \"kubernetes.io/projected/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-kube-api-access-bjs44\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.605362 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.605307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:07.737057 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:07.736951    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"]
Jul 09 18:00:07.739190 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:00:07.739161    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ea98d3_6e34_4dbb_ab0f_4c8f1c720ed5.slice/crio-16536b18b36165fc7d26a6cfc2f0629999420adca04d3a73f1d1fd1782fd0ede WatchSource:0}: Error finding container 16536b18b36165fc7d26a6cfc2f0629999420adca04d3a73f1d1fd1782fd0ede: Status 404 returned error can't find the container with id 16536b18b36165fc7d26a6cfc2f0629999420adca04d3a73f1d1fd1782fd0ede
Jul 09 18:00:08.302973 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:08.302940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph" event={"ID":"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5","Type":"ContainerStarted","Data":"23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a"}
Jul 09 18:00:08.303330 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:08.302993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph" event={"ID":"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5","Type":"ContainerStarted","Data":"16536b18b36165fc7d26a6cfc2f0629999420adca04d3a73f1d1fd1782fd0ede"}
Jul 09 18:00:08.323485 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:08.323440    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph" podStartSLOduration=1.3233997180000001 podStartE2EDuration="1.323399718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:00:08.323260685 +0000 UTC m=+1242.513620604" watchObservedRunningTime="2026-07-09 18:00:08.323399718 +0000 UTC m=+1242.513759662"
Jul 09 18:00:08.606355 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:08.606280    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:17.606059 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:17.606029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:17.617747 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:17.617719    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:18.360233 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:18.360167    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:00:26.254320 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:26.254294    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 18:00:51.710102 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:00:51.710068    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"]
Jul 09 18:01:06.730007 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:06.729965    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph" podUID="44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" containerName="main" containerID="cri-o://23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a" gracePeriod=45
Jul 09 18:01:06.971725 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:06.971696    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:01:07.031789 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.031763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tls-certs\") pod \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") "
Jul 09 18:01:07.031929 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.031811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-dshm\") pod \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") "
Jul 09 18:01:07.031929 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.031865    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-model-cache\") pod \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") "
Jul 09 18:01:07.031929 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.031894    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-home\") pod \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") "
Jul 09 18:01:07.032087 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.031929    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tmp-dir\") pod \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") "
Jul 09 18:01:07.032087 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.031966    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjs44\" (UniqueName: \"kubernetes.io/projected/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-kube-api-access-bjs44\") pod \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\" (UID: \"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5\") "
Jul 09 18:01:07.032188 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.032151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-model-cache" (OuterVolumeSpecName: "model-cache") pod "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" (UID: "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:07.032244 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.032203    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" (UID: "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:07.032244 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.032218    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-home" (OuterVolumeSpecName: "home") pod "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" (UID: "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:07.033885 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.033814    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-dshm" (OuterVolumeSpecName: "dshm") pod "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" (UID: "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:07.034157 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.033933    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" (UID: "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:01:07.034157 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.033935    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-kube-api-access-bjs44" (OuterVolumeSpecName: "kube-api-access-bjs44") pod "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" (UID: "44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5"). InnerVolumeSpecName "kube-api-access-bjs44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:01:07.133397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.133371    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:07.133397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.133393    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:07.133524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.133402    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:07.133524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.133410    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:07.133524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.133434    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:07.133524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.133443    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjs44\" (UniqueName: \"kubernetes.io/projected/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5-kube-api-access-bjs44\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:07.547762 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.547736    2577 generic.go:356] "Generic (PLEG): container finished" podID="44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" containerID="23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a" exitCode=0
Jul 09 18:01:07.547915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.547791    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"
Jul 09 18:01:07.547915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.547829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph" event={"ID":"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5","Type":"ContainerDied","Data":"23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a"}
Jul 09 18:01:07.547915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.547898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph" event={"ID":"44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5","Type":"ContainerDied","Data":"16536b18b36165fc7d26a6cfc2f0629999420adca04d3a73f1d1fd1782fd0ede"}
Jul 09 18:01:07.548090 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.547921    2577 scope.go:117] "RemoveContainer" containerID="23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a"
Jul 09 18:01:07.556717 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.556695    2577 scope.go:117] "RemoveContainer" containerID="23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a"
Jul 09 18:01:07.556993 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:01:07.556974    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a\": container with ID starting with 23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a not found: ID does not exist" containerID="23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a"
Jul 09 18:01:07.557064 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.557001    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a"} err="failed to get container status \"23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a\": rpc error: code = NotFound desc = could not find container \"23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a\": container with ID starting with 23abe26660d393c60fa1093b871ce3db740ad70ddc1dc250e8ade3eed041349a not found: ID does not exist"
Jul 09 18:01:07.571651 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.571630    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"]
Jul 09 18:01:07.573801 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:07.573783    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fmz2ph"]
Jul 09 18:01:08.417630 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:08.417605    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" path="/var/lib/kubelet/pods/44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5/volumes"
Jul 09 18:01:12.136411 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:12.136380    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"]
Jul 09 18:01:27.156475 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.156423    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv" podUID="5bccce0d-2b49-4d99-934b-ae0d7a946379" containerName="main" containerID="cri-o://b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac" gracePeriod=45
Jul 09 18:01:27.400867 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.400820    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 18:01:27.480510 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480448    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bccce0d-2b49-4d99-934b-ae0d7a946379-tls-certs\") pod \"5bccce0d-2b49-4d99-934b-ae0d7a946379\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") "
Jul 09 18:01:27.480510 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-tmp-dir\") pod \"5bccce0d-2b49-4d99-934b-ae0d7a946379\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") "
Jul 09 18:01:27.480670 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480545    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-home\") pod \"5bccce0d-2b49-4d99-934b-ae0d7a946379\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") "
Jul 09 18:01:27.480670 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480560    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-dshm\") pod \"5bccce0d-2b49-4d99-934b-ae0d7a946379\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") "
Jul 09 18:01:27.480670 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4psjh\" (UniqueName: \"kubernetes.io/projected/5bccce0d-2b49-4d99-934b-ae0d7a946379-kube-api-access-4psjh\") pod \"5bccce0d-2b49-4d99-934b-ae0d7a946379\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") "
Jul 09 18:01:27.480670 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480609    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-model-cache\") pod \"5bccce0d-2b49-4d99-934b-ae0d7a946379\" (UID: \"5bccce0d-2b49-4d99-934b-ae0d7a946379\") "
Jul 09 18:01:27.480930 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480806    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-home" (OuterVolumeSpecName: "home") pod "5bccce0d-2b49-4d99-934b-ae0d7a946379" (UID: "5bccce0d-2b49-4d99-934b-ae0d7a946379"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:27.480930 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5bccce0d-2b49-4d99-934b-ae0d7a946379" (UID: "5bccce0d-2b49-4d99-934b-ae0d7a946379"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:27.481012 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.480954    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-model-cache" (OuterVolumeSpecName: "model-cache") pod "5bccce0d-2b49-4d99-934b-ae0d7a946379" (UID: "5bccce0d-2b49-4d99-934b-ae0d7a946379"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:27.482640 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.482608    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bccce0d-2b49-4d99-934b-ae0d7a946379-kube-api-access-4psjh" (OuterVolumeSpecName: "kube-api-access-4psjh") pod "5bccce0d-2b49-4d99-934b-ae0d7a946379" (UID: "5bccce0d-2b49-4d99-934b-ae0d7a946379"). InnerVolumeSpecName "kube-api-access-4psjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:01:27.482640 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.482614    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-dshm" (OuterVolumeSpecName: "dshm") pod "5bccce0d-2b49-4d99-934b-ae0d7a946379" (UID: "5bccce0d-2b49-4d99-934b-ae0d7a946379"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:27.482769 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.482641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bccce0d-2b49-4d99-934b-ae0d7a946379-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5bccce0d-2b49-4d99-934b-ae0d7a946379" (UID: "5bccce0d-2b49-4d99-934b-ae0d7a946379"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:01:27.581911 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.581888    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:27.581911 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.581908    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:27.582060 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.581916    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:27.582060 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.581926    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4psjh\" (UniqueName: \"kubernetes.io/projected/5bccce0d-2b49-4d99-934b-ae0d7a946379-kube-api-access-4psjh\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:27.582060 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.581937    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bccce0d-2b49-4d99-934b-ae0d7a946379-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:27.582060 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.581945    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bccce0d-2b49-4d99-934b-ae0d7a946379-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:27.628883 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.628856    2577 generic.go:356] "Generic (PLEG): container finished" podID="5bccce0d-2b49-4d99-934b-ae0d7a946379" containerID="b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac" exitCode=0
Jul 09 18:01:27.628969 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.628899    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"
Jul 09 18:01:27.628969 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.628913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv" event={"ID":"5bccce0d-2b49-4d99-934b-ae0d7a946379","Type":"ContainerDied","Data":"b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac"}
Jul 09 18:01:27.628969 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.628965    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv" event={"ID":"5bccce0d-2b49-4d99-934b-ae0d7a946379","Type":"ContainerDied","Data":"8c59811df57b71b1660933714940e2059975ce5a666d6af6339ecf44ba01270b"}
Jul 09 18:01:27.629066 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.629016    2577 scope.go:117] "RemoveContainer" containerID="b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac"
Jul 09 18:01:27.637401 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.637387    2577 scope.go:117] "RemoveContainer" containerID="b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac"
Jul 09 18:01:27.637631 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:01:27.637615    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac\": container with ID starting with b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac not found: ID does not exist" containerID="b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac"
Jul 09 18:01:27.637695 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.637643    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac"} err="failed to get container status \"b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac\": rpc error: code = NotFound desc = could not find container \"b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac\": container with ID starting with b48e8e2d7a85cc6ea0abe81c07ccac89d39b4f935295cf25db3365c20e5550ac not found: ID does not exist"
Jul 09 18:01:27.658759 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.658729    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"]
Jul 09 18:01:27.666734 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:27.666715    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-router-managed-test-llm-4b931143-kserve-bd545d56-6ghzv"]
Jul 09 18:01:28.418008 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:28.417980    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bccce0d-2b49-4d99-934b-ae0d7a946379" path="/var/lib/kubelet/pods/5bccce0d-2b49-4d99-934b-ae0d7a946379/volumes"
Jul 09 18:01:35.637951 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.637919    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"]
Jul 09 18:01:35.638439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.638190    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" containerName="main"
Jul 09 18:01:35.638439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.638200    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" containerName="main"
Jul 09 18:01:35.638439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.638220    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bccce0d-2b49-4d99-934b-ae0d7a946379" containerName="main"
Jul 09 18:01:35.638439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.638226    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bccce0d-2b49-4d99-934b-ae0d7a946379" containerName="main"
Jul 09 18:01:35.638439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.638270    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ea98d3-6e34-4dbb-ab0f-4c8f1c720ed5" containerName="main"
Jul 09 18:01:35.638439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.638278    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bccce0d-2b49-4d99-934b-ae0d7a946379" containerName="main"
Jul 09 18:01:35.642456 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.642437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.645096 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.645075    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 09 18:01:35.646190 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.646172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-b56b5\""
Jul 09 18:01:35.656057 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.656037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"]
Jul 09 18:01:35.738383 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.738355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43002fe7-954b-4ee0-862d-9f82e9bb292b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.738510 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.738394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.738510 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.738424    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.738605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.738530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2bs\" (UniqueName: \"kubernetes.io/projected/43002fe7-954b-4ee0-862d-9f82e9bb292b-kube-api-access-kp2bs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.839223 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.839195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.839329 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.839225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.839329 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.839260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2bs\" (UniqueName: \"kubernetes.io/projected/43002fe7-954b-4ee0-862d-9f82e9bb292b-kube-api-access-kp2bs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.839329 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.839301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43002fe7-954b-4ee0-862d-9f82e9bb292b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.839551 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.839532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.839616 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.839598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.841617 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.841589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43002fe7-954b-4ee0-862d-9f82e9bb292b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.848770 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.848746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2bs\" (UniqueName: \"kubernetes.io/projected/43002fe7-954b-4ee0-862d-9f82e9bb292b-kube-api-access-kp2bs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:35.950723 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:35.950667    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:36.093108 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:36.093085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"]
Jul 09 18:01:36.095272 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:01:36.095244    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43002fe7_954b_4ee0_862d_9f82e9bb292b.slice/crio-89d4207284aff7d09b7d115dff6a07156c7d022fb6cf633089aab8f203d7677e WatchSource:0}: Error finding container 89d4207284aff7d09b7d115dff6a07156c7d022fb6cf633089aab8f203d7677e: Status 404 returned error can't find the container with id 89d4207284aff7d09b7d115dff6a07156c7d022fb6cf633089aab8f203d7677e
Jul 09 18:01:36.669552 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:36.669515    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" event={"ID":"43002fe7-954b-4ee0-862d-9f82e9bb292b","Type":"ContainerStarted","Data":"54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18"}
Jul 09 18:01:36.670028 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:36.669566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" event={"ID":"43002fe7-954b-4ee0-862d-9f82e9bb292b","Type":"ContainerStarted","Data":"89d4207284aff7d09b7d115dff6a07156c7d022fb6cf633089aab8f203d7677e"}
Jul 09 18:01:37.674607 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:37.674576    2577 generic.go:356] "Generic (PLEG): container finished" podID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerID="54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18" exitCode=0
Jul 09 18:01:37.675004 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:37.674624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" event={"ID":"43002fe7-954b-4ee0-862d-9f82e9bb292b","Type":"ContainerDied","Data":"54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18"}
Jul 09 18:01:38.680547 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:38.680504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" event={"ID":"43002fe7-954b-4ee0-862d-9f82e9bb292b","Type":"ContainerStarted","Data":"d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f"}
Jul 09 18:01:38.681004 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:38.680652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:01:38.707964 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:38.707907    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" podStartSLOduration=3.04177031 podStartE2EDuration="3.70789439s" totalImagesPullingTime="0s" totalInitContainerRuntime="666.12408ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:01:38.707714411 +0000 UTC m=+1332.898074327" watchObservedRunningTime="2026-07-09 18:01:38.70789439 +0000 UTC m=+1332.898254304"
Jul 09 18:01:57.102449 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.102372    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"]
Jul 09 18:01:57.111187 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:01:57.111155    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:01:57.111187 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:01:57.111187 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:01:57.111187 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c" cmd=["/bin/sleep","15"]
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.111200    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" message="" err=<
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:01:57.111212    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerName="main" containerID="cri-o://fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c"
Jul 09 18:01:57.111464 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.111257    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerName="main" containerID="cri-o://fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c" gracePeriod=60
Jul 09 18:01:57.356694 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.356642    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 18:01:57.506483 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.506438    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tls-certs\") pod \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") "
Jul 09 18:01:57.506626 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.506543    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6v8h\" (UniqueName: \"kubernetes.io/projected/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kube-api-access-m6v8h\") pod \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") "
Jul 09 18:01:57.506626 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.506593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kserve-provision-location\") pod \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") "
Jul 09 18:01:57.506626 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.506621    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tokenizer-uds\") pod \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\" (UID: \"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4\") "
Jul 09 18:01:57.506925 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.506896    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" (UID: "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:57.507272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.507251    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" (UID: "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:01:57.508360 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.508328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" (UID: "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:01:57.508459 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.508441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kube-api-access-m6v8h" (OuterVolumeSpecName: "kube-api-access-m6v8h") pod "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" (UID: "33f8ddcd-132c-4e06-b5d3-3f34d2e275a4"). InnerVolumeSpecName "kube-api-access-m6v8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:01:57.607952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.607883    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:57.607952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.607919    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:57.607952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.607928    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6v8h\" (UniqueName: \"kubernetes.io/projected/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kube-api-access-m6v8h\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:57.607952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.607938    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:01:57.764786 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.764747    2577 generic.go:356] "Generic (PLEG): container finished" podID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerID="fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c" exitCode=0
Jul 09 18:01:57.765004 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.764802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" event={"ID":"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4","Type":"ContainerDied","Data":"fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c"}
Jul 09 18:01:57.765004 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.764818    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"
Jul 09 18:01:57.765004 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.764878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm" event={"ID":"33f8ddcd-132c-4e06-b5d3-3f34d2e275a4","Type":"ContainerDied","Data":"63049cecb2c3b4994441c9cf33fb7f07852b17a8ec663b02817dab04434f07e9"}
Jul 09 18:01:57.765004 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.764911    2577 scope.go:117] "RemoveContainer" containerID="fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c"
Jul 09 18:01:57.775393 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.775359    2577 scope.go:117] "RemoveContainer" containerID="41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb"
Jul 09 18:01:57.785641 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.785618    2577 scope.go:117] "RemoveContainer" containerID="fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c"
Jul 09 18:01:57.785921 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:01:57.785893    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c\": container with ID starting with fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c not found: ID does not exist" containerID="fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c"
Jul 09 18:01:57.785988 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.785934    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c"} err="failed to get container status \"fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c\": rpc error: code = NotFound desc = could not find container \"fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c\": container with ID starting with fd3d834e8e39573069a3d3e238a2e89b409596f69c4d3347990a8afed683791c not found: ID does not exist"
Jul 09 18:01:57.785988 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.785959    2577 scope.go:117] "RemoveContainer" containerID="41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb"
Jul 09 18:01:57.786266 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:01:57.786243    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb\": container with ID starting with 41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb not found: ID does not exist" containerID="41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb"
Jul 09 18:01:57.786320 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.786277    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb"} err="failed to get container status \"41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb\": rpc error: code = NotFound desc = could not find container \"41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb\": container with ID starting with 41d13da3d49b3155ed65e8a3e05c397aaae8f12911f45c74688bf087564328eb not found: ID does not exist"
Jul 09 18:01:57.791802 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.791777    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"]
Jul 09 18:01:57.796981 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:57.796961    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-565d6d8849-8sgkm"]
Jul 09 18:01:58.417601 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:01:58.417567    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" path="/var/lib/kubelet/pods/33f8ddcd-132c-4e06-b5d3-3f34d2e275a4/volumes"
Jul 09 18:02:09.687081 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:09.687052    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:02:34.391120 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.391083    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"]
Jul 09 18:02:34.391631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.391360    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerName="main"
Jul 09 18:02:34.391631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.391372    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerName="main"
Jul 09 18:02:34.391631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.391382    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerName="storage-initializer"
Jul 09 18:02:34.391631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.391387    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerName="storage-initializer"
Jul 09 18:02:34.391631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.391449    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f8ddcd-132c-4e06-b5d3-3f34d2e275a4" containerName="main"
Jul 09 18:02:34.395783 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.395759    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.398421 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.398398    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Jul 09 18:02:34.406947 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.406929    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"]
Jul 09 18:02:34.476769 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.476741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.476896 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.476775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.476896 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.476791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.477031 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.476944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdpsj\" (UniqueName: \"kubernetes.io/projected/2c790703-e99d-4207-8b05-6b6c2bba97d4-kube-api-access-tdpsj\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.477031 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.477000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.477031 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.477026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c790703-e99d-4207-8b05-6b6c2bba97d4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.577612 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.577571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c790703-e99d-4207-8b05-6b6c2bba97d4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.577797 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.577626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.577797 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.577657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.577797 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.577671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.577797 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.577746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdpsj\" (UniqueName: \"kubernetes.io/projected/2c790703-e99d-4207-8b05-6b6c2bba97d4-kube-api-access-tdpsj\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.577797 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.577790    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.578157 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.578134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.578223 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.578168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.578223 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.578186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.580136 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.580113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.580295 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.580277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c790703-e99d-4207-8b05-6b6c2bba97d4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.587185 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.587151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdpsj\" (UniqueName: \"kubernetes.io/projected/2c790703-e99d-4207-8b05-6b6c2bba97d4-kube-api-access-tdpsj\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.706220 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.706153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:34.844889 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.844849    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"]
Jul 09 18:02:34.847443 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:02:34.847415    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c790703_e99d_4207_8b05_6b6c2bba97d4.slice/crio-f2f59b506128abbd7de47f7780c38e532a44bae6ab220ae70b878ea393f8ec2b WatchSource:0}: Error finding container f2f59b506128abbd7de47f7780c38e532a44bae6ab220ae70b878ea393f8ec2b: Status 404 returned error can't find the container with id f2f59b506128abbd7de47f7780c38e532a44bae6ab220ae70b878ea393f8ec2b
Jul 09 18:02:34.926981 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.926941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2" event={"ID":"2c790703-e99d-4207-8b05-6b6c2bba97d4","Type":"ContainerStarted","Data":"918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370"}
Jul 09 18:02:34.927108 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.927025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2" event={"ID":"2c790703-e99d-4207-8b05-6b6c2bba97d4","Type":"ContainerStarted","Data":"f2f59b506128abbd7de47f7780c38e532a44bae6ab220ae70b878ea393f8ec2b"}
Jul 09 18:02:34.952774 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:34.952730    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2" podStartSLOduration=0.952714117 podStartE2EDuration="952.714117ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:02:34.950426676 +0000 UTC m=+1389.140786592" watchObservedRunningTime="2026-07-09 18:02:34.952714117 +0000 UTC m=+1389.143074031"
Jul 09 18:02:35.707130 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:35.707091    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:44.706747 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:44.706670    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:44.719097 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:44.719076    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:44.983009 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:44.982930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:02:51.620053 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:02:51.620024    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"]
Jul 09 18:03:06.639303 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:06.639259    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2" podUID="2c790703-e99d-4207-8b05-6b6c2bba97d4" containerName="main" containerID="cri-o://918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370" gracePeriod=45
Jul 09 18:03:06.877507 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:06.877480    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:03:07.029477 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029449    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c790703-e99d-4207-8b05-6b6c2bba97d4-tls-certs\") pod \"2c790703-e99d-4207-8b05-6b6c2bba97d4\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") "
Jul 09 18:03:07.029635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029491    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-home\") pod \"2c790703-e99d-4207-8b05-6b6c2bba97d4\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") "
Jul 09 18:03:07.029635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029509    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-dshm\") pod \"2c790703-e99d-4207-8b05-6b6c2bba97d4\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") "
Jul 09 18:03:07.029635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029528    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-model-cache\") pod \"2c790703-e99d-4207-8b05-6b6c2bba97d4\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") "
Jul 09 18:03:07.029635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029573    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdpsj\" (UniqueName: \"kubernetes.io/projected/2c790703-e99d-4207-8b05-6b6c2bba97d4-kube-api-access-tdpsj\") pod \"2c790703-e99d-4207-8b05-6b6c2bba97d4\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") "
Jul 09 18:03:07.029635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029592    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-tmp-dir\") pod \"2c790703-e99d-4207-8b05-6b6c2bba97d4\" (UID: \"2c790703-e99d-4207-8b05-6b6c2bba97d4\") "
Jul 09 18:03:07.029955 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029728    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-home" (OuterVolumeSpecName: "home") pod "2c790703-e99d-4207-8b05-6b6c2bba97d4" (UID: "2c790703-e99d-4207-8b05-6b6c2bba97d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:07.029955 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c790703-e99d-4207-8b05-6b6c2bba97d4" (UID: "2c790703-e99d-4207-8b05-6b6c2bba97d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:07.029955 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.029934    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c790703-e99d-4207-8b05-6b6c2bba97d4" (UID: "2c790703-e99d-4207-8b05-6b6c2bba97d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:07.031671 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.031646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-dshm" (OuterVolumeSpecName: "dshm") pod "2c790703-e99d-4207-8b05-6b6c2bba97d4" (UID: "2c790703-e99d-4207-8b05-6b6c2bba97d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:07.031774 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.031701    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c790703-e99d-4207-8b05-6b6c2bba97d4-kube-api-access-tdpsj" (OuterVolumeSpecName: "kube-api-access-tdpsj") pod "2c790703-e99d-4207-8b05-6b6c2bba97d4" (UID: "2c790703-e99d-4207-8b05-6b6c2bba97d4"). InnerVolumeSpecName "kube-api-access-tdpsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:03:07.031827 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.031774    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c790703-e99d-4207-8b05-6b6c2bba97d4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c790703-e99d-4207-8b05-6b6c2bba97d4" (UID: "2c790703-e99d-4207-8b05-6b6c2bba97d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:03:07.067474 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.067450    2577 generic.go:356] "Generic (PLEG): container finished" podID="2c790703-e99d-4207-8b05-6b6c2bba97d4" containerID="918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370" exitCode=0
Jul 09 18:03:07.067585 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.067492    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"
Jul 09 18:03:07.067585 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.067495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2" event={"ID":"2c790703-e99d-4207-8b05-6b6c2bba97d4","Type":"ContainerDied","Data":"918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370"}
Jul 09 18:03:07.067585 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.067544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2" event={"ID":"2c790703-e99d-4207-8b05-6b6c2bba97d4","Type":"ContainerDied","Data":"f2f59b506128abbd7de47f7780c38e532a44bae6ab220ae70b878ea393f8ec2b"}
Jul 09 18:03:07.067585 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.067568    2577 scope.go:117] "RemoveContainer" containerID="918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370"
Jul 09 18:03:07.078761 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.078743    2577 scope.go:117] "RemoveContainer" containerID="918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370"
Jul 09 18:03:07.079152 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:03:07.079135    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370\": container with ID starting with 918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370 not found: ID does not exist" containerID="918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370"
Jul 09 18:03:07.079227 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.079165    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370"} err="failed to get container status \"918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370\": rpc error: code = NotFound desc = could not find container \"918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370\": container with ID starting with 918d283624a56e9584b5ad5dc6c63b71d21e27ffd85b0f360e9b395b3d702370 not found: ID does not exist"
Jul 09 18:03:07.094657 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.094634    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"]
Jul 09 18:03:07.097812 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.097790    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-66874c76d4khzx2"]
Jul 09 18:03:07.130941 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.130923    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdpsj\" (UniqueName: \"kubernetes.io/projected/2c790703-e99d-4207-8b05-6b6c2bba97d4-kube-api-access-tdpsj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:07.131019 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.130942    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:07.131019 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.130953    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c790703-e99d-4207-8b05-6b6c2bba97d4-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:07.131019 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.130962    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:07.131019 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.130969    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:07.131019 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:07.130977    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c790703-e99d-4207-8b05-6b6c2bba97d4-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:08.417818 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:08.417776    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c790703-e99d-4207-8b05-6b6c2bba97d4" path="/var/lib/kubelet/pods/2c790703-e99d-4207-8b05-6b6c2bba97d4/volumes"
Jul 09 18:03:17.228203 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.228169    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"]
Jul 09 18:03:17.228606 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.228588    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c790703-e99d-4207-8b05-6b6c2bba97d4" containerName="main"
Jul 09 18:03:17.228650 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.228609    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c790703-e99d-4207-8b05-6b6c2bba97d4" containerName="main"
Jul 09 18:03:17.228706 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.228695    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c790703-e99d-4207-8b05-6b6c2bba97d4" containerName="main"
Jul 09 18:03:17.232990 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.232970    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.236638 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.236609    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-5zs9m\""
Jul 09 18:03:17.237827 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.237809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 09 18:03:17.244569 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.244550    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"]
Jul 09 18:03:17.408033 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.407994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45344cb6-57f0-4f92-99b7-0d7215d16736-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.408226 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.408044    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.408226 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.408107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.408226 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.408142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdnd\" (UniqueName: \"kubernetes.io/projected/45344cb6-57f0-4f92-99b7-0d7215d16736-kube-api-access-gmdnd\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.509489 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.509451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45344cb6-57f0-4f92-99b7-0d7215d16736-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.509656 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.509500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.509656 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.509524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.509656 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.509550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdnd\" (UniqueName: \"kubernetes.io/projected/45344cb6-57f0-4f92-99b7-0d7215d16736-kube-api-access-gmdnd\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.509933 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.509905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-kserve-provision-location\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.510049 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.509973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-tokenizer-uds\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.511920 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.511892    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45344cb6-57f0-4f92-99b7-0d7215d16736-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.518136 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.518117    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdnd\" (UniqueName: \"kubernetes.io/projected/45344cb6-57f0-4f92-99b7-0d7215d16736-kube-api-access-gmdnd\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") " pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.542033 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.542014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:17.673963 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:17.673902    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"]
Jul 09 18:03:17.676228 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:03:17.676199    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45344cb6_57f0_4f92_99b7_0d7215d16736.slice/crio-2115ba5241aaf13b272c10a3bcd793dacc55980f2c73cccfd4bb11dfa10e274d WatchSource:0}: Error finding container 2115ba5241aaf13b272c10a3bcd793dacc55980f2c73cccfd4bb11dfa10e274d: Status 404 returned error can't find the container with id 2115ba5241aaf13b272c10a3bcd793dacc55980f2c73cccfd4bb11dfa10e274d
Jul 09 18:03:18.116272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:18.116185    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" event={"ID":"45344cb6-57f0-4f92-99b7-0d7215d16736","Type":"ContainerStarted","Data":"20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db"}
Jul 09 18:03:18.116272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:18.116240    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" event={"ID":"45344cb6-57f0-4f92-99b7-0d7215d16736","Type":"ContainerStarted","Data":"2115ba5241aaf13b272c10a3bcd793dacc55980f2c73cccfd4bb11dfa10e274d"}
Jul 09 18:03:19.122189 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:19.122153    2577 generic.go:356] "Generic (PLEG): container finished" podID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerID="20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db" exitCode=0
Jul 09 18:03:19.122189 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:19.122199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" event={"ID":"45344cb6-57f0-4f92-99b7-0d7215d16736","Type":"ContainerDied","Data":"20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db"}
Jul 09 18:03:19.612769 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:19.612735    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"]
Jul 09 18:03:19.623047 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:03:19.623010    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:03:19.623047 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:03:19.623047 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:03:19.623047 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f" cmd=["/bin/sleep","15"]
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:19.623058    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" message="" err=<
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:03:19.623071    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="main" containerID="cri-o://d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f"
Jul 09 18:03:19.623374 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:19.623111    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="main" containerID="cri-o://d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f" gracePeriod=60
Jul 09 18:03:19.685503 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:03:19.685478    2577 logging.go:55] [core] [Channel #316 SubChannel #317]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:9003", ServerName: "10.133.0.38:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:9003: connect: connection refused"
Jul 09 18:03:19.867400 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:19.867342    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:03:20.030032 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.029999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-tokenizer-uds\") pod \"43002fe7-954b-4ee0-862d-9f82e9bb292b\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") "
Jul 09 18:03:20.030187 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.030069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43002fe7-954b-4ee0-862d-9f82e9bb292b-tls-certs\") pod \"43002fe7-954b-4ee0-862d-9f82e9bb292b\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") "
Jul 09 18:03:20.030187 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.030097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kp2bs\" (UniqueName: \"kubernetes.io/projected/43002fe7-954b-4ee0-862d-9f82e9bb292b-kube-api-access-kp2bs\") pod \"43002fe7-954b-4ee0-862d-9f82e9bb292b\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") "
Jul 09 18:03:20.030187 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.030165    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-kserve-provision-location\") pod \"43002fe7-954b-4ee0-862d-9f82e9bb292b\" (UID: \"43002fe7-954b-4ee0-862d-9f82e9bb292b\") "
Jul 09 18:03:20.030350 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.030218    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "43002fe7-954b-4ee0-862d-9f82e9bb292b" (UID: "43002fe7-954b-4ee0-862d-9f82e9bb292b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:20.030350 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.030343    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:20.030987 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.030963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "43002fe7-954b-4ee0-862d-9f82e9bb292b" (UID: "43002fe7-954b-4ee0-862d-9f82e9bb292b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:03:20.032015 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.031994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43002fe7-954b-4ee0-862d-9f82e9bb292b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43002fe7-954b-4ee0-862d-9f82e9bb292b" (UID: "43002fe7-954b-4ee0-862d-9f82e9bb292b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:03:20.032170 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.032150    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43002fe7-954b-4ee0-862d-9f82e9bb292b-kube-api-access-kp2bs" (OuterVolumeSpecName: "kube-api-access-kp2bs") pod "43002fe7-954b-4ee0-862d-9f82e9bb292b" (UID: "43002fe7-954b-4ee0-862d-9f82e9bb292b"). InnerVolumeSpecName "kube-api-access-kp2bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:03:20.128217 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.128158    2577 generic.go:356] "Generic (PLEG): container finished" podID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerID="d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f" exitCode=0
Jul 09 18:03:20.128573 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.128219    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"
Jul 09 18:03:20.128573 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.128241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" event={"ID":"43002fe7-954b-4ee0-862d-9f82e9bb292b","Type":"ContainerDied","Data":"d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f"}
Jul 09 18:03:20.128573 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.128308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" event={"ID":"43002fe7-954b-4ee0-862d-9f82e9bb292b","Type":"ContainerDied","Data":"89d4207284aff7d09b7d115dff6a07156c7d022fb6cf633089aab8f203d7677e"}
Jul 09 18:03:20.128573 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.128340    2577 scope.go:117] "RemoveContainer" containerID="d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f"
Jul 09 18:03:20.130370 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.130344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" event={"ID":"45344cb6-57f0-4f92-99b7-0d7215d16736","Type":"ContainerStarted","Data":"cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d"}
Jul 09 18:03:20.130493 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.130402    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:03:20.130663 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.130645    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43002fe7-954b-4ee0-862d-9f82e9bb292b-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:20.130729 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.130665    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2bs\" (UniqueName: \"kubernetes.io/projected/43002fe7-954b-4ee0-862d-9f82e9bb292b-kube-api-access-kp2bs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:20.130729 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.130675    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/43002fe7-954b-4ee0-862d-9f82e9bb292b-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:03:20.137049 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.137031    2577 scope.go:117] "RemoveContainer" containerID="54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18"
Jul 09 18:03:20.143980 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.143961    2577 scope.go:117] "RemoveContainer" containerID="d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f"
Jul 09 18:03:20.144233 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:03:20.144215    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f\": container with ID starting with d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f not found: ID does not exist" containerID="d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f"
Jul 09 18:03:20.144272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.144242    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f"} err="failed to get container status \"d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f\": rpc error: code = NotFound desc = could not find container \"d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f\": container with ID starting with d5bfffa15715f538416dc46ed383705fc85a12971254e79f8a3abf3ed55a5b7f not found: ID does not exist"
Jul 09 18:03:20.144272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.144259    2577 scope.go:117] "RemoveContainer" containerID="54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18"
Jul 09 18:03:20.144502 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:03:20.144485    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18\": container with ID starting with 54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18 not found: ID does not exist" containerID="54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18"
Jul 09 18:03:20.144550 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.144509    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18"} err="failed to get container status \"54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18\": rpc error: code = NotFound desc = could not find container \"54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18\": container with ID starting with 54e50b432db286066c9c658c1feb49612a41e6eda9087e212845f1848c4bbd18 not found: ID does not exist"
Jul 09 18:03:20.160704 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.160667    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" podStartSLOduration=2.530757006 podStartE2EDuration="3.160654244s" totalImagesPullingTime="0s" totalInitContainerRuntime="629.897238ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:20.159407239 +0000 UTC m=+1434.349767253" watchObservedRunningTime="2026-07-09 18:03:20.160654244 +0000 UTC m=+1434.351014158"
Jul 09 18:03:20.172396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.172375    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"]
Jul 09 18:03:20.176125 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.176104    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk"]
Jul 09 18:03:20.417690 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.417631    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" path="/var/lib/kubelet/pods/43002fe7-954b-4ee0-862d-9f82e9bb292b/volumes"
Jul 09 18:03:20.686001 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:20.685865    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schedcjmk" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.133.0.38:9003\" within 1s: context deadline exceeded"
Jul 09 18:03:20.686174 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:03:20.685906    2577 logging.go:55] [core] [Channel #316 SubChannel #317]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.38:9003", ServerName: "10.133.0.38:9003", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.38:9003: operation was canceled"
Jul 09 18:03:44.156312 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.156282    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"]
Jul 09 18:03:44.156739 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.156605    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="storage-initializer"
Jul 09 18:03:44.156739 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.156617    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="storage-initializer"
Jul 09 18:03:44.156739 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.156625    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="main"
Jul 09 18:03:44.156739 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.156630    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="main"
Jul 09 18:03:44.156739 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.156686    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="43002fe7-954b-4ee0-862d-9f82e9bb292b" containerName="main"
Jul 09 18:03:44.161124 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.161104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.164020 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.163994    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 09 18:03:44.164137 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.164028    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-nnmc9\""
Jul 09 18:03:44.171779 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.171759    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"]
Jul 09 18:03:44.187081 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.187054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c5e7685-a76b-42fa-8048-fb682f709bfe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.187200 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.187091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5e7685-a76b-42fa-8048-fb682f709bfe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.187200 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.187130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6c5e7685-a76b-42fa-8048-fb682f709bfe-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.187200 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.187156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q75mr\" (UniqueName: \"kubernetes.io/projected/6c5e7685-a76b-42fa-8048-fb682f709bfe-kube-api-access-q75mr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.288277 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.288242    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c5e7685-a76b-42fa-8048-fb682f709bfe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.288419 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.288285    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5e7685-a76b-42fa-8048-fb682f709bfe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.288490 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.288406    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6c5e7685-a76b-42fa-8048-fb682f709bfe-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.288490 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.288442    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q75mr\" (UniqueName: \"kubernetes.io/projected/6c5e7685-a76b-42fa-8048-fb682f709bfe-kube-api-access-q75mr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.288609 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.288589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c5e7685-a76b-42fa-8048-fb682f709bfe-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.288691 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.288670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6c5e7685-a76b-42fa-8048-fb682f709bfe-tokenizer-uds\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.290683 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.290662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c5e7685-a76b-42fa-8048-fb682f709bfe-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.297219 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.297200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q75mr\" (UniqueName: \"kubernetes.io/projected/6c5e7685-a76b-42fa-8048-fb682f709bfe-kube-api-access-q75mr\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr\" (UID: \"6c5e7685-a76b-42fa-8048-fb682f709bfe\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.471413 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.471354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:44.606934 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:44.606909    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"]
Jul 09 18:03:44.608575 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:03:44.608539    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5e7685_a76b_42fa_8048_fb682f709bfe.slice/crio-1aa15dd278a5c94078c1ba35511dc03798817dda317f0ec04a783161ccfa63c3 WatchSource:0}: Error finding container 1aa15dd278a5c94078c1ba35511dc03798817dda317f0ec04a783161ccfa63c3: Status 404 returned error can't find the container with id 1aa15dd278a5c94078c1ba35511dc03798817dda317f0ec04a783161ccfa63c3
Jul 09 18:03:45.237006 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:45.236966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr" event={"ID":"6c5e7685-a76b-42fa-8048-fb682f709bfe","Type":"ContainerStarted","Data":"7c5534907c6b8b36b7dab582ccd1c0f45aef51f6ce9eec0661418a78e9e27949"}
Jul 09 18:03:45.237370 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:45.237022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr" event={"ID":"6c5e7685-a76b-42fa-8048-fb682f709bfe","Type":"ContainerStarted","Data":"1aa15dd278a5c94078c1ba35511dc03798817dda317f0ec04a783161ccfa63c3"}
Jul 09 18:03:46.242708 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:46.242671    2577 generic.go:356] "Generic (PLEG): container finished" podID="6c5e7685-a76b-42fa-8048-fb682f709bfe" containerID="7c5534907c6b8b36b7dab582ccd1c0f45aef51f6ce9eec0661418a78e9e27949" exitCode=0
Jul 09 18:03:46.243101 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:46.242725    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr" event={"ID":"6c5e7685-a76b-42fa-8048-fb682f709bfe","Type":"ContainerDied","Data":"7c5534907c6b8b36b7dab582ccd1c0f45aef51f6ce9eec0661418a78e9e27949"}
Jul 09 18:03:47.249194 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:47.249155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr" event={"ID":"6c5e7685-a76b-42fa-8048-fb682f709bfe","Type":"ContainerStarted","Data":"2b93fe0fb95f6ba7f790f7a8a5dc30e40027c5fc604b3e284f5856f20b916b0a"}
Jul 09 18:03:47.249598 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:47.249272    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:03:47.290401 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:47.290345    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr" podStartSLOduration=2.653115036 podStartE2EDuration="3.29032351s" totalImagesPullingTime="0s" totalInitContainerRuntime="637.208474ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:47.287242157 +0000 UTC m=+1461.477602071" watchObservedRunningTime="2026-07-09 18:03:47.29032351 +0000 UTC m=+1461.480683426"
Jul 09 18:03:51.137619 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:03:51.137590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:04:18.254683 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:04:18.254655    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr"
Jul 09 18:04:26.299928 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:04:26.299901    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:04:26.307918 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:04:26.307898    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:05:18.380719 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:18.380689    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:05:18.388515 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:18.388494    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:05:22.004721 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.003926    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"]
Jul 09 18:05:22.015140 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:05:22.015105    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:05:22.015140 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:05:22.015140 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:05:22.015140 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d" cmd=["/bin/sleep","15"]
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.015155    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" message="" err=<
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:05:22.015166    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerName="main" containerID="cri-o://cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d"
Jul 09 18:05:22.015413 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.015210    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerName="main" containerID="cri-o://cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d" gracePeriod=60
Jul 09 18:05:22.256173 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.256115    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:05:22.336644 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.336619    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-kserve-provision-location\") pod \"45344cb6-57f0-4f92-99b7-0d7215d16736\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") "
Jul 09 18:05:22.336755 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.336656    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmdnd\" (UniqueName: \"kubernetes.io/projected/45344cb6-57f0-4f92-99b7-0d7215d16736-kube-api-access-gmdnd\") pod \"45344cb6-57f0-4f92-99b7-0d7215d16736\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") "
Jul 09 18:05:22.336755 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.336746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45344cb6-57f0-4f92-99b7-0d7215d16736-tls-certs\") pod \"45344cb6-57f0-4f92-99b7-0d7215d16736\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") "
Jul 09 18:05:22.336863 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.336773    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-tokenizer-uds\") pod \"45344cb6-57f0-4f92-99b7-0d7215d16736\" (UID: \"45344cb6-57f0-4f92-99b7-0d7215d16736\") "
Jul 09 18:05:22.337124 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.337087    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "45344cb6-57f0-4f92-99b7-0d7215d16736" (UID: "45344cb6-57f0-4f92-99b7-0d7215d16736"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:22.337408 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.337386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45344cb6-57f0-4f92-99b7-0d7215d16736" (UID: "45344cb6-57f0-4f92-99b7-0d7215d16736"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:05:22.338737 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.338711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45344cb6-57f0-4f92-99b7-0d7215d16736-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45344cb6-57f0-4f92-99b7-0d7215d16736" (UID: "45344cb6-57f0-4f92-99b7-0d7215d16736"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:05:22.338820 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.338800    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45344cb6-57f0-4f92-99b7-0d7215d16736-kube-api-access-gmdnd" (OuterVolumeSpecName: "kube-api-access-gmdnd") pod "45344cb6-57f0-4f92-99b7-0d7215d16736" (UID: "45344cb6-57f0-4f92-99b7-0d7215d16736"). InnerVolumeSpecName "kube-api-access-gmdnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:22.438236 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.438208    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45344cb6-57f0-4f92-99b7-0d7215d16736-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.438236 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.438236    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.438398 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.438250    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45344cb6-57f0-4f92-99b7-0d7215d16736-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.438398 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.438264    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmdnd\" (UniqueName: \"kubernetes.io/projected/45344cb6-57f0-4f92-99b7-0d7215d16736-kube-api-access-gmdnd\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:05:22.662114 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.662038    2577 generic.go:356] "Generic (PLEG): container finished" podID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerID="cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d" exitCode=0
Jul 09 18:05:22.662114 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.662082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" event={"ID":"45344cb6-57f0-4f92-99b7-0d7215d16736","Type":"ContainerDied","Data":"cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d"}
Jul 09 18:05:22.662114 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.662094    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"
Jul 09 18:05:22.662332 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.662121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm" event={"ID":"45344cb6-57f0-4f92-99b7-0d7215d16736","Type":"ContainerDied","Data":"2115ba5241aaf13b272c10a3bcd793dacc55980f2c73cccfd4bb11dfa10e274d"}
Jul 09 18:05:22.662332 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.662143    2577 scope.go:117] "RemoveContainer" containerID="cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d"
Jul 09 18:05:22.670333 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.670321    2577 scope.go:117] "RemoveContainer" containerID="20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db"
Jul 09 18:05:22.678054 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.678018    2577 scope.go:117] "RemoveContainer" containerID="cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d"
Jul 09 18:05:22.678355 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:05:22.678335    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d\": container with ID starting with cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d not found: ID does not exist" containerID="cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d"
Jul 09 18:05:22.678402 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.678366    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d"} err="failed to get container status \"cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d\": rpc error: code = NotFound desc = could not find container \"cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d\": container with ID starting with cc194a0996b6f0801d8285344add7769d122411e4ee8ddaa2a6ff844f1c2c50d not found: ID does not exist"
Jul 09 18:05:22.678402 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.678389    2577 scope.go:117] "RemoveContainer" containerID="20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db"
Jul 09 18:05:22.678655 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:05:22.678634    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db\": container with ID starting with 20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db not found: ID does not exist" containerID="20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db"
Jul 09 18:05:22.678721 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.678659    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db"} err="failed to get container status \"20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db\": rpc error: code = NotFound desc = could not find container \"20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db\": container with ID starting with 20824537d9c295291b0212ac37e7e3c09b9ef7eab2b930c6579970df174969db not found: ID does not exist"
Jul 09 18:05:22.679249 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.679232    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"]
Jul 09 18:05:22.683313 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:22.683291    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schexshdm"]
Jul 09 18:05:23.454290 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.454262    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"]
Jul 09 18:05:23.454683 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.454580    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerName="main"
Jul 09 18:05:23.454683 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.454591    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerName="main"
Jul 09 18:05:23.454683 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.454613    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerName="storage-initializer"
Jul 09 18:05:23.454683 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.454619    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerName="storage-initializer"
Jul 09 18:05:23.454683 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.454670    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" containerName="main"
Jul 09 18:05:23.458823 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.458800    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:23.465744 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.465675    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"]
Jul 09 18:05:23.543428 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.543403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmf7\" (UniqueName: \"kubernetes.io/projected/6f2f5fd1-2d8a-4656-b792-ed15ed54539c-kube-api-access-dlmf7\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:23.643783 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.643755    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmf7\" (UniqueName: \"kubernetes.io/projected/6f2f5fd1-2d8a-4656-b792-ed15ed54539c-kube-api-access-dlmf7\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:23.652295 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:23.652267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmf7\" (UniqueName: \"kubernetes.io/projected/6f2f5fd1-2d8a-4656-b792-ed15ed54539c-kube-api-access-dlmf7\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:24.417304 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:24.417270    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45344cb6-57f0-4f92-99b7-0d7215d16736" path="/var/lib/kubelet/pods/45344cb6-57f0-4f92-99b7-0d7215d16736/volumes"
Jul 09 18:05:26.163233 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.163188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.169293 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.169268    2577 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.263685 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.263660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.263807 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.263753    2577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.267231 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.267211    2577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") DevicePath \"csi-865c1b7864345fca6c1b33e4b028fc235615dd119089aa7e2da66591f7cbbc73\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.507917 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.507882    2577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/17b411dca5d48cfac4cdaa10248b91d7ef13accf2569b96b481df6f6ce336a23/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.520930 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.520906    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"e2e-pvc-model-download-q2skc\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.768397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.768341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:26.891271 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:26.891240    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"]
Jul 09 18:05:26.894857 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:05:26.894815    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f2f5fd1_2d8a_4656_b792_ed15ed54539c.slice/crio-a8cf7ceec2ebf1ac84b96c3cf6dda5755b9870b7c2bc3e8da85afb4a53ffec32 WatchSource:0}: Error finding container a8cf7ceec2ebf1ac84b96c3cf6dda5755b9870b7c2bc3e8da85afb4a53ffec32: Status 404 returned error can't find the container with id a8cf7ceec2ebf1ac84b96c3cf6dda5755b9870b7c2bc3e8da85afb4a53ffec32
Jul 09 18:05:27.691590 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:27.691547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc" event={"ID":"6f2f5fd1-2d8a-4656-b792-ed15ed54539c","Type":"ContainerStarted","Data":"d76c2d5fa65166ab0bbfab9ab31c8871d51e0a38229a7dd3250b0bcf2b5df7d4"}
Jul 09 18:05:27.692110 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:27.691596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc" event={"ID":"6f2f5fd1-2d8a-4656-b792-ed15ed54539c","Type":"ContainerStarted","Data":"a8cf7ceec2ebf1ac84b96c3cf6dda5755b9870b7c2bc3e8da85afb4a53ffec32"}
Jul 09 18:05:27.712645 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:27.712584    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc" podStartSLOduration=7.712556341 podStartE2EDuration="7.712556341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:05:27.711766613 +0000 UTC m=+1561.902126530" watchObservedRunningTime="2026-07-09 18:05:27.712556341 +0000 UTC m=+1561.902916256"
Jul 09 18:05:33.717269 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:33.717242    2577 generic.go:356] "Generic (PLEG): container finished" podID="6f2f5fd1-2d8a-4656-b792-ed15ed54539c" containerID="d76c2d5fa65166ab0bbfab9ab31c8871d51e0a38229a7dd3250b0bcf2b5df7d4" exitCode=0
Jul 09 18:05:33.717620 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:33.717279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc" event={"ID":"6f2f5fd1-2d8a-4656-b792-ed15ed54539c","Type":"ContainerDied","Data":"d76c2d5fa65166ab0bbfab9ab31c8871d51e0a38229a7dd3250b0bcf2b5df7d4"}
Jul 09 18:05:34.858506 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:34.858481    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:34.929495 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:34.929468    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") pod \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") "
Jul 09 18:05:34.929622 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:34.929539    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlmf7\" (UniqueName: \"kubernetes.io/projected/6f2f5fd1-2d8a-4656-b792-ed15ed54539c-kube-api-access-dlmf7\") pod \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\" (UID: \"6f2f5fd1-2d8a-4656-b792-ed15ed54539c\") "
Jul 09 18:05:34.931768 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:34.931736    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2f5fd1-2d8a-4656-b792-ed15ed54539c-kube-api-access-dlmf7" (OuterVolumeSpecName: "kube-api-access-dlmf7") pod "6f2f5fd1-2d8a-4656-b792-ed15ed54539c" (UID: "6f2f5fd1-2d8a-4656-b792-ed15ed54539c"). InnerVolumeSpecName "kube-api-access-dlmf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:05:34.933226 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:34.933205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf" (OuterVolumeSpecName: "model-storage") pod "6f2f5fd1-2d8a-4656-b792-ed15ed54539c" (UID: "6f2f5fd1-2d8a-4656-b792-ed15ed54539c"). InnerVolumeSpecName "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 18:05:35.030787 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:35.030765    2577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-141-44.ec2.internal\" "
Jul 09 18:05:35.030787 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:35.030788    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlmf7\" (UniqueName: \"kubernetes.io/projected/6f2f5fd1-2d8a-4656-b792-ed15ed54539c-kube-api-access-dlmf7\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:05:35.136686 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:35.136669    2577 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf") on node "ip-10-0-141-44.ec2.internal"
Jul 09 18:05:35.232124 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:35.232102    2577 reconciler_common.go:299] "Volume detached for volume \"pvc-d2189ca3-9cab-4233-b265-8aa73aeb27a0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-086cd812ba4a253cf\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"csi-865c1b7864345fca6c1b33e4b028fc235615dd119089aa7e2da66591f7cbbc73\""
Jul 09 18:05:35.729050 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:35.729024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc" event={"ID":"6f2f5fd1-2d8a-4656-b792-ed15ed54539c","Type":"ContainerDied","Data":"a8cf7ceec2ebf1ac84b96c3cf6dda5755b9870b7c2bc3e8da85afb4a53ffec32"}
Jul 09 18:05:35.729229 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:35.729054    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-q2skc"
Jul 09 18:05:35.729229 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:05:35.729060    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8cf7ceec2ebf1ac84b96c3cf6dda5755b9870b7c2bc3e8da85afb4a53ffec32"
Jul 09 18:06:08.930353 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.930324    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"]
Jul 09 18:06:08.930939 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.930647    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2f5fd1-2d8a-4656-b792-ed15ed54539c" containerName="storage-initializer"
Jul 09 18:06:08.930939 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.930660    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2f5fd1-2d8a-4656-b792-ed15ed54539c" containerName="storage-initializer"
Jul 09 18:06:08.930939 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.930719    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2f5fd1-2d8a-4656-b792-ed15ed54539c" containerName="storage-initializer"
Jul 09 18:06:08.933568 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.933549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:08.937517 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.937499    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 09 18:06:08.937625 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.937543    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-96vz6\""
Jul 09 18:06:08.945272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:08.945251    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"]
Jul 09 18:06:09.058165 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.058140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9s7n\" (UniqueName: \"kubernetes.io/projected/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-kube-api-access-c9s7n\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.058286 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.058172    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.058286 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.058201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.158682 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.158657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9s7n\" (UniqueName: \"kubernetes.io/projected/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-kube-api-access-c9s7n\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.158811 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.158702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.158811 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.158750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.159126 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.159107    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tokenizer-uds\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.161123 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.161104    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.167016 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.166995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9s7n\" (UniqueName: \"kubernetes.io/projected/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-kube-api-access-c9s7n\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") " pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.242624 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.242573    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.374307 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.374247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"]
Jul 09 18:06:09.376490 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:06:09.376461    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfc5fb9_6c43_4c15_8f49_cb9423076a46.slice/crio-4a29f2cecd13b2113424d0ccc5bb0a0e34f338acf6fd35dc68ecb7fef91a9368 WatchSource:0}: Error finding container 4a29f2cecd13b2113424d0ccc5bb0a0e34f338acf6fd35dc68ecb7fef91a9368: Status 404 returned error can't find the container with id 4a29f2cecd13b2113424d0ccc5bb0a0e34f338acf6fd35dc68ecb7fef91a9368
Jul 09 18:06:09.865853 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.865799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" event={"ID":"7dfc5fb9-6c43-4c15-8f49-cb9423076a46","Type":"ContainerStarted","Data":"7a6852d856991f21c35075fd2d48e0fdf8250b4b1603cd4654e24c39732a1f14"}
Jul 09 18:06:09.866040 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.865918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" event={"ID":"7dfc5fb9-6c43-4c15-8f49-cb9423076a46","Type":"ContainerStarted","Data":"4a29f2cecd13b2113424d0ccc5bb0a0e34f338acf6fd35dc68ecb7fef91a9368"}
Jul 09 18:06:09.866040 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.865968    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:09.887439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:09.887381    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" podStartSLOduration=1.887361719 podStartE2EDuration="1.887361719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:09.884972649 +0000 UTC m=+1604.075332565" watchObservedRunningTime="2026-07-09 18:06:09.887361719 +0000 UTC m=+1604.077721635"
Jul 09 18:06:18.063741 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.063668    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"]
Jul 09 18:06:18.067007 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.066986    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.070807 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.070779    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 09 18:06:18.070938 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.070832    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-epp-sa-dockercfg-zxqd8\""
Jul 09 18:06:18.081906 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.081872    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"]
Jul 09 18:06:18.235193 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.235163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.235364 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.235200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.235364 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.235245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.235364 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.235314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhv2k\" (UniqueName: \"kubernetes.io/projected/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kube-api-access-mhv2k\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.336634 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.336566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhv2k\" (UniqueName: \"kubernetes.io/projected/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kube-api-access-mhv2k\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.336634 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.336598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.336634 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.336619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.336864 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.336663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.337100 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.337080    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.337147 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.337099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tokenizer-uds\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.338919 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.338897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.344872 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.344829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhv2k\" (UniqueName: \"kubernetes.io/projected/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kube-api-access-mhv2k\") pod \"custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.389603 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.389579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:18.535221 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.535184    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"]
Jul 09 18:06:18.539019 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:06:18.538987    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27994d3f_5c4a_4091_b584_3b6a0de4e52e.slice/crio-af4839b1d8e3721a0aa07cfaa4cdc406a785f813d8ea1d97ec7799c118a81a1c WatchSource:0}: Error finding container af4839b1d8e3721a0aa07cfaa4cdc406a785f813d8ea1d97ec7799c118a81a1c: Status 404 returned error can't find the container with id af4839b1d8e3721a0aa07cfaa4cdc406a785f813d8ea1d97ec7799c118a81a1c
Jul 09 18:06:18.906467 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.906391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" event={"ID":"27994d3f-5c4a-4091-b584-3b6a0de4e52e","Type":"ContainerStarted","Data":"f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85"}
Jul 09 18:06:18.906467 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:18.906440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" event={"ID":"27994d3f-5c4a-4091-b584-3b6a0de4e52e","Type":"ContainerStarted","Data":"af4839b1d8e3721a0aa07cfaa4cdc406a785f813d8ea1d97ec7799c118a81a1c"}
Jul 09 18:06:19.913383 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:19.913342    2577 generic.go:356] "Generic (PLEG): container finished" podID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerID="f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85" exitCode=0
Jul 09 18:06:19.913383 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:19.913385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" event={"ID":"27994d3f-5c4a-4091-b584-3b6a0de4e52e","Type":"ContainerDied","Data":"f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85"}
Jul 09 18:06:20.920563 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:20.920528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" event={"ID":"27994d3f-5c4a-4091-b584-3b6a0de4e52e","Type":"ContainerStarted","Data":"80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105"}
Jul 09 18:06:20.920924 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:20.920647    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:06:20.945386 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:20.945346    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" podStartSLOduration=2.294979985 podStartE2EDuration="2.945331841s" totalImagesPullingTime="0s" totalInitContainerRuntime="650.351856ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:06:20.942934733 +0000 UTC m=+1615.133294672" watchObservedRunningTime="2026-07-09 18:06:20.945331841 +0000 UTC m=+1615.135691756"
Jul 09 18:06:40.871497 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:40.871468    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:06:51.926805 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:06:51.926778    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:08:47.406239 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.406208    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"]
Jul 09 18:08:47.415100 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:08:47.415060    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:08:47.415100 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:08:47.415100 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:08:47.415100 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="7a6852d856991f21c35075fd2d48e0fdf8250b4b1603cd4654e24c39732a1f14" cmd=["/bin/sleep","15"]
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.415117    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" message="" err=<
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:08:47.415129    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" podUID="7dfc5fb9-6c43-4c15-8f49-cb9423076a46" containerName="main" containerID="cri-o://7a6852d856991f21c35075fd2d48e0fdf8250b4b1603cd4654e24c39732a1f14"
Jul 09 18:08:47.415369 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.415164    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" podUID="7dfc5fb9-6c43-4c15-8f49-cb9423076a46" containerName="main" containerID="cri-o://7a6852d856991f21c35075fd2d48e0fdf8250b4b1603cd4654e24c39732a1f14" gracePeriod=60
Jul 09 18:08:47.600305 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.600279    2577 generic.go:356] "Generic (PLEG): container finished" podID="7dfc5fb9-6c43-4c15-8f49-cb9423076a46" containerID="7a6852d856991f21c35075fd2d48e0fdf8250b4b1603cd4654e24c39732a1f14" exitCode=0
Jul 09 18:08:47.600429 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.600346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" event={"ID":"7dfc5fb9-6c43-4c15-8f49-cb9423076a46","Type":"ContainerDied","Data":"7a6852d856991f21c35075fd2d48e0fdf8250b4b1603cd4654e24c39732a1f14"}
Jul 09 18:08:47.670060 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.670039    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:08:47.768342 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.768315    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tokenizer-uds\") pod \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") "
Jul 09 18:08:47.768456 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.768354    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9s7n\" (UniqueName: \"kubernetes.io/projected/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-kube-api-access-c9s7n\") pod \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") "
Jul 09 18:08:47.768456 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.768398    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tls-certs\") pod \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\" (UID: \"7dfc5fb9-6c43-4c15-8f49-cb9423076a46\") "
Jul 09 18:08:47.768617 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.768591    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7dfc5fb9-6c43-4c15-8f49-cb9423076a46" (UID: "7dfc5fb9-6c43-4c15-8f49-cb9423076a46"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:47.770445 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.770418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-kube-api-access-c9s7n" (OuterVolumeSpecName: "kube-api-access-c9s7n") pod "7dfc5fb9-6c43-4c15-8f49-cb9423076a46" (UID: "7dfc5fb9-6c43-4c15-8f49-cb9423076a46"). InnerVolumeSpecName "kube-api-access-c9s7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:08:47.770552 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.770464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7dfc5fb9-6c43-4c15-8f49-cb9423076a46" (UID: "7dfc5fb9-6c43-4c15-8f49-cb9423076a46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:08:47.868994 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.868967    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:08:47.868994 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.868987    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9s7n\" (UniqueName: \"kubernetes.io/projected/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-kube-api-access-c9s7n\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:08:47.868994 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:47.868997    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7dfc5fb9-6c43-4c15-8f49-cb9423076a46-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:08:48.606166 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:48.606134    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"
Jul 09 18:08:48.606642 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:48.606169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf" event={"ID":"7dfc5fb9-6c43-4c15-8f49-cb9423076a46","Type":"ContainerDied","Data":"4a29f2cecd13b2113424d0ccc5bb0a0e34f338acf6fd35dc68ecb7fef91a9368"}
Jul 09 18:08:48.606642 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:48.606225    2577 scope.go:117] "RemoveContainer" containerID="7a6852d856991f21c35075fd2d48e0fdf8250b4b1603cd4654e24c39732a1f14"
Jul 09 18:08:48.626128 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:48.626100    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"]
Jul 09 18:08:48.631782 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:48.631761    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schelrzbf"]
Jul 09 18:08:50.417188 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:50.417153    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dfc5fb9-6c43-4c15-8f49-cb9423076a46" path="/var/lib/kubelet/pods/7dfc5fb9-6c43-4c15-8f49-cb9423076a46/volumes"
Jul 09 18:08:54.200899 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.200867    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"]
Jul 09 18:08:54.209858 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:08:54.209792    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:08:54.209858 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:08:54.209858 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:08:54.209858 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105" cmd=["/bin/sleep","15"]
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.209869    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" message="" err=<
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:08:54.209882    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerName="main" containerID="cri-o://80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105"
Jul 09 18:08:54.210096 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.209934    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerName="main" containerID="cri-o://80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105" gracePeriod=60
Jul 09 18:08:54.465034 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.464963    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:08:54.519110 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.519069    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tokenizer-uds\") pod \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") "
Jul 09 18:08:54.519263 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.519129    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tls-certs\") pod \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") "
Jul 09 18:08:54.519263 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.519151    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhv2k\" (UniqueName: \"kubernetes.io/projected/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kube-api-access-mhv2k\") pod \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") "
Jul 09 18:08:54.519263 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.519205    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kserve-provision-location\") pod \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\" (UID: \"27994d3f-5c4a-4091-b584-3b6a0de4e52e\") "
Jul 09 18:08:54.519448 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.519283    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "27994d3f-5c4a-4091-b584-3b6a0de4e52e" (UID: "27994d3f-5c4a-4091-b584-3b6a0de4e52e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:54.519448 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.519384    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:08:54.519871 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.519852    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27994d3f-5c4a-4091-b584-3b6a0de4e52e" (UID: "27994d3f-5c4a-4091-b584-3b6a0de4e52e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:08:54.521135 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.521107    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27994d3f-5c4a-4091-b584-3b6a0de4e52e" (UID: "27994d3f-5c4a-4091-b584-3b6a0de4e52e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:08:54.521230 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.521174    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kube-api-access-mhv2k" (OuterVolumeSpecName: "kube-api-access-mhv2k") pod "27994d3f-5c4a-4091-b584-3b6a0de4e52e" (UID: "27994d3f-5c4a-4091-b584-3b6a0de4e52e"). InnerVolumeSpecName "kube-api-access-mhv2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:08:54.620278 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.620243    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:08:54.620278 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.620277    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27994d3f-5c4a-4091-b584-3b6a0de4e52e-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:08:54.620422 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.620294    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhv2k\" (UniqueName: \"kubernetes.io/projected/27994d3f-5c4a-4091-b584-3b6a0de4e52e-kube-api-access-mhv2k\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:08:54.633629 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.633599    2577 generic.go:356] "Generic (PLEG): container finished" podID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerID="80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105" exitCode=0
Jul 09 18:08:54.633752 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.633638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" event={"ID":"27994d3f-5c4a-4091-b584-3b6a0de4e52e","Type":"ContainerDied","Data":"80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105"}
Jul 09 18:08:54.633752 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.633661    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"
Jul 09 18:08:54.633752 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.633691    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq" event={"ID":"27994d3f-5c4a-4091-b584-3b6a0de4e52e","Type":"ContainerDied","Data":"af4839b1d8e3721a0aa07cfaa4cdc406a785f813d8ea1d97ec7799c118a81a1c"}
Jul 09 18:08:54.633752 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.633718    2577 scope.go:117] "RemoveContainer" containerID="80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105"
Jul 09 18:08:54.642259 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.642239    2577 scope.go:117] "RemoveContainer" containerID="f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85"
Jul 09 18:08:54.650065 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.650045    2577 scope.go:117] "RemoveContainer" containerID="80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105"
Jul 09 18:08:54.650336 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:08:54.650320    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105\": container with ID starting with 80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105 not found: ID does not exist" containerID="80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105"
Jul 09 18:08:54.650385 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.650342    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105"} err="failed to get container status \"80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105\": rpc error: code = NotFound desc = could not find container \"80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105\": container with ID starting with 80a405a7b83e20e433ffc29506bd14622e7fc3cbb0b1dac43315a810b3226105 not found: ID does not exist"
Jul 09 18:08:54.650385 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.650358    2577 scope.go:117] "RemoveContainer" containerID="f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85"
Jul 09 18:08:54.650603 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:08:54.650588    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85\": container with ID starting with f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85 not found: ID does not exist" containerID="f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85"
Jul 09 18:08:54.650651 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.650608    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85"} err="failed to get container status \"f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85\": rpc error: code = NotFound desc = could not find container \"f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85\": container with ID starting with f8ec85bf39fcaee34e29daefc3e820983b0211174ffcaad90d37b0788d50da85 not found: ID does not exist"
Jul 09 18:08:54.654769 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.654744    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"]
Jul 09 18:08:54.659456 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:54.659417    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-router-scheduler-5955866cj96wq"]
Jul 09 18:08:56.417351 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:08:56.417314    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" path="/var/lib/kubelet/pods/27994d3f-5c4a-4091-b584-3b6a0de4e52e/volumes"
Jul 09 18:09:13.024904 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.024870    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"]
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025147    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfc5fb9-6c43-4c15-8f49-cb9423076a46" containerName="main"
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025157    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfc5fb9-6c43-4c15-8f49-cb9423076a46" containerName="main"
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025166    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerName="main"
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025172    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerName="main"
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025185    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerName="storage-initializer"
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025191    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerName="storage-initializer"
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025238    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27994d3f-5c4a-4091-b584-3b6a0de4e52e" containerName="main"
Jul 09 18:09:13.025473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.025247    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfc5fb9-6c43-4c15-8f49-cb9423076a46" containerName="main"
Jul 09 18:09:13.028185 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.028164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.032344 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.032317    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 09 18:09:13.032481 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.032463    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-5t8mf\""
Jul 09 18:09:13.040983 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.040961    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"]
Jul 09 18:09:13.062915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.062891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590b41e5-a840-4a8d-8b6a-78969f365f99-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.063007 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.062946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgdkv\" (UniqueName: \"kubernetes.io/projected/590b41e5-a840-4a8d-8b6a-78969f365f99-kube-api-access-zgdkv\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.063007 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.062976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/590b41e5-a840-4a8d-8b6a-78969f365f99-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.163597 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.163568    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590b41e5-a840-4a8d-8b6a-78969f365f99-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.163684 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.163620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgdkv\" (UniqueName: \"kubernetes.io/projected/590b41e5-a840-4a8d-8b6a-78969f365f99-kube-api-access-zgdkv\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.163684 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.163655    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/590b41e5-a840-4a8d-8b6a-78969f365f99-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.164045 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.164028    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/590b41e5-a840-4a8d-8b6a-78969f365f99-tokenizer-uds\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.166424 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.166388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590b41e5-a840-4a8d-8b6a-78969f365f99-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.172694 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.172671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgdkv\" (UniqueName: \"kubernetes.io/projected/590b41e5-a840-4a8d-8b6a-78969f365f99-kube-api-access-zgdkv\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") " pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.337635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.337555    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.476670 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.476644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"]
Jul 09 18:09:13.478592 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:09:13.478565    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590b41e5_a840_4a8d_8b6a_78969f365f99.slice/crio-8ef356b03128b320fcfa56af19710ef6bdfd46fdf3677e66ef4c90be27c14ada WatchSource:0}: Error finding container 8ef356b03128b320fcfa56af19710ef6bdfd46fdf3677e66ef4c90be27c14ada: Status 404 returned error can't find the container with id 8ef356b03128b320fcfa56af19710ef6bdfd46fdf3677e66ef4c90be27c14ada
Jul 09 18:09:13.720172 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.720083    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" event={"ID":"590b41e5-a840-4a8d-8b6a-78969f365f99","Type":"ContainerStarted","Data":"17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448"}
Jul 09 18:09:13.720172 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.720148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" event={"ID":"590b41e5-a840-4a8d-8b6a-78969f365f99","Type":"ContainerStarted","Data":"8ef356b03128b320fcfa56af19710ef6bdfd46fdf3677e66ef4c90be27c14ada"}
Jul 09 18:09:13.720362 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.720176    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:09:13.738496 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:13.738446    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" podStartSLOduration=0.738430953 podStartE2EDuration="738.430953ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:09:13.737874144 +0000 UTC m=+1787.928234059" watchObservedRunningTime="2026-07-09 18:09:13.738430953 +0000 UTC m=+1787.928790868"
Jul 09 18:09:26.326659 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:26.326631    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:09:26.332012 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:26.331987    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:09:44.725973 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:09:44.725942    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:10:04.671083 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.671057    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"]
Jul 09 18:10:04.674237 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.674219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.677074 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.677053    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-epp-sa-dockercfg-mpxm8\""
Jul 09 18:10:04.678093 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.678068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 09 18:10:04.686875 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.686854    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"]
Jul 09 18:10:04.748764 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.748734    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.748904 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.748780    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96221ac9-3eab-4343-9023-26f7a8c1e875-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.748904 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.748832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7x9h\" (UniqueName: \"kubernetes.io/projected/96221ac9-3eab-4343-9023-26f7a8c1e875-kube-api-access-b7x9h\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.748989 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.748929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.850180 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.850104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96221ac9-3eab-4343-9023-26f7a8c1e875-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.850301 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.850239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7x9h\" (UniqueName: \"kubernetes.io/projected/96221ac9-3eab-4343-9023-26f7a8c1e875-kube-api-access-b7x9h\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.850301 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.850268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.850413 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.850301    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.850631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.850613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-tokenizer-uds\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.850706 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.850628    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-kserve-provision-location\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.852511 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.852494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96221ac9-3eab-4343-9023-26f7a8c1e875-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.859025 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.858998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7x9h\" (UniqueName: \"kubernetes.io/projected/96221ac9-3eab-4343-9023-26f7a8c1e875-kube-api-access-b7x9h\") pod \"router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:04.983396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:04.983374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:05.118055 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:05.118027    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"]
Jul 09 18:10:05.119662 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:10:05.119634    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96221ac9_3eab_4343_9023_26f7a8c1e875.slice/crio-d0eb37b919bd20471674ed4b271a6682bdd936b624ab944bdf3bfde0dca41b07 WatchSource:0}: Error finding container d0eb37b919bd20471674ed4b271a6682bdd936b624ab944bdf3bfde0dca41b07: Status 404 returned error can't find the container with id d0eb37b919bd20471674ed4b271a6682bdd936b624ab944bdf3bfde0dca41b07
Jul 09 18:10:05.949364 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:05.949329    2577 generic.go:356] "Generic (PLEG): container finished" podID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerID="dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12" exitCode=0
Jul 09 18:10:05.949726 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:05.949413    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" event={"ID":"96221ac9-3eab-4343-9023-26f7a8c1e875","Type":"ContainerDied","Data":"dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12"}
Jul 09 18:10:05.949726 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:05.949476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" event={"ID":"96221ac9-3eab-4343-9023-26f7a8c1e875","Type":"ContainerStarted","Data":"d0eb37b919bd20471674ed4b271a6682bdd936b624ab944bdf3bfde0dca41b07"}
Jul 09 18:10:06.955194 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:06.955155    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" event={"ID":"96221ac9-3eab-4343-9023-26f7a8c1e875","Type":"ContainerStarted","Data":"f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2"}
Jul 09 18:10:06.955590 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:06.955281    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:10:06.977811 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:06.977766    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" podStartSLOduration=2.334862019 podStartE2EDuration="2.97775293s" totalImagesPullingTime="0s" totalInitContainerRuntime="642.890911ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:10:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:10:06.977349729 +0000 UTC m=+1841.167709654" watchObservedRunningTime="2026-07-09 18:10:06.97775293 +0000 UTC m=+1841.168112845"
Jul 09 18:10:37.961263 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:10:37.961231    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:12:31.207850 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.207644    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"]
Jul 09 18:12:31.217693 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:12:31.217659    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:12:31.217693 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:12:31.217693 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:12:31.217693 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2" cmd=["/bin/sleep","15"]
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.217705    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" message="" err=<
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:12:31.217717    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerName="main" containerID="cri-o://f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2"
Jul 09 18:12:31.218052 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.217762    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerName="main" containerID="cri-o://f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2" gracePeriod=60
Jul 09 18:12:31.468025 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.467966    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:12:31.572270 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.572240    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96221ac9-3eab-4343-9023-26f7a8c1e875-tls-certs\") pod \"96221ac9-3eab-4343-9023-26f7a8c1e875\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") "
Jul 09 18:12:31.572414 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.572318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-tokenizer-uds\") pod \"96221ac9-3eab-4343-9023-26f7a8c1e875\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") "
Jul 09 18:12:31.572414 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.572355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7x9h\" (UniqueName: \"kubernetes.io/projected/96221ac9-3eab-4343-9023-26f7a8c1e875-kube-api-access-b7x9h\") pod \"96221ac9-3eab-4343-9023-26f7a8c1e875\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") "
Jul 09 18:12:31.572414 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.572372    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-kserve-provision-location\") pod \"96221ac9-3eab-4343-9023-26f7a8c1e875\" (UID: \"96221ac9-3eab-4343-9023-26f7a8c1e875\") "
Jul 09 18:12:31.572626 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.572601    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "96221ac9-3eab-4343-9023-26f7a8c1e875" (UID: "96221ac9-3eab-4343-9023-26f7a8c1e875"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:31.573102 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.573071    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "96221ac9-3eab-4343-9023-26f7a8c1e875" (UID: "96221ac9-3eab-4343-9023-26f7a8c1e875"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:31.574439 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.574407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96221ac9-3eab-4343-9023-26f7a8c1e875-kube-api-access-b7x9h" (OuterVolumeSpecName: "kube-api-access-b7x9h") pod "96221ac9-3eab-4343-9023-26f7a8c1e875" (UID: "96221ac9-3eab-4343-9023-26f7a8c1e875"). InnerVolumeSpecName "kube-api-access-b7x9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:31.574609 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.574587    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96221ac9-3eab-4343-9023-26f7a8c1e875-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96221ac9-3eab-4343-9023-26f7a8c1e875" (UID: "96221ac9-3eab-4343-9023-26f7a8c1e875"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:12:31.618678 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.618654    2577 generic.go:356] "Generic (PLEG): container finished" podID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerID="f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2" exitCode=0
Jul 09 18:12:31.618751 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.618694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" event={"ID":"96221ac9-3eab-4343-9023-26f7a8c1e875","Type":"ContainerDied","Data":"f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2"}
Jul 09 18:12:31.618751 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.618705    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"
Jul 09 18:12:31.618751 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.618734    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb" event={"ID":"96221ac9-3eab-4343-9023-26f7a8c1e875","Type":"ContainerDied","Data":"d0eb37b919bd20471674ed4b271a6682bdd936b624ab944bdf3bfde0dca41b07"}
Jul 09 18:12:31.618898 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.618753    2577 scope.go:117] "RemoveContainer" containerID="f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2"
Jul 09 18:12:31.628159 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.628141    2577 scope.go:117] "RemoveContainer" containerID="dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12"
Jul 09 18:12:31.635584 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.635566    2577 scope.go:117] "RemoveContainer" containerID="f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2"
Jul 09 18:12:31.635901 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:12:31.635818    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2\": container with ID starting with f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2 not found: ID does not exist" containerID="f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2"
Jul 09 18:12:31.635901 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.635858    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2"} err="failed to get container status \"f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2\": rpc error: code = NotFound desc = could not find container \"f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2\": container with ID starting with f7893791615dae719b3c6b71f1ccaeef2005a040f5324c9b3686303c8be9d1e2 not found: ID does not exist"
Jul 09 18:12:31.635901 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.635876    2577 scope.go:117] "RemoveContainer" containerID="dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12"
Jul 09 18:12:31.636110 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:12:31.636090    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12\": container with ID starting with dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12 not found: ID does not exist" containerID="dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12"
Jul 09 18:12:31.636150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.636116    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12"} err="failed to get container status \"dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12\": rpc error: code = NotFound desc = could not find container \"dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12\": container with ID starting with dbd1c976b3cc9df6cfdd057dd5e9700536af4a6d6a5a01e6ae91efad5f934d12 not found: ID does not exist"
Jul 09 18:12:31.640578 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.640559    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"]
Jul 09 18:12:31.646557 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.646537    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-router-scheduler-5b9d496c6c-9xznb"]
Jul 09 18:12:31.673261 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.673237    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:12:31.673261 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.673259    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7x9h\" (UniqueName: \"kubernetes.io/projected/96221ac9-3eab-4343-9023-26f7a8c1e875-kube-api-access-b7x9h\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:12:31.673381 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.673269    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/96221ac9-3eab-4343-9023-26f7a8c1e875-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:12:31.673381 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:31.673279    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96221ac9-3eab-4343-9023-26f7a8c1e875-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:12:32.418175 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:32.418146    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" path="/var/lib/kubelet/pods/96221ac9-3eab-4343-9023-26f7a8c1e875/volumes"
Jul 09 18:12:40.879495 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:40.879461    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"]
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:12:40.888831    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448" cmd=["/bin/sleep","15"]
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:40.888895    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" message="" err=<
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:12:40.888904 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:12:40.889342 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:12:40.888906    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:12:40.889342 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:12:40.889342 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:12:40.889342 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" podUID="590b41e5-a840-4a8d-8b6a-78969f365f99" containerName="main" containerID="cri-o://17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448"
Jul 09 18:12:40.889342 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:40.888953    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" podUID="590b41e5-a840-4a8d-8b6a-78969f365f99" containerName="main" containerID="cri-o://17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448" gracePeriod=60
Jul 09 18:12:41.142873 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.142773    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:12:41.248159 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.248133    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgdkv\" (UniqueName: \"kubernetes.io/projected/590b41e5-a840-4a8d-8b6a-78969f365f99-kube-api-access-zgdkv\") pod \"590b41e5-a840-4a8d-8b6a-78969f365f99\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") "
Jul 09 18:12:41.248328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.248174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/590b41e5-a840-4a8d-8b6a-78969f365f99-tokenizer-uds\") pod \"590b41e5-a840-4a8d-8b6a-78969f365f99\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") "
Jul 09 18:12:41.248328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.248216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590b41e5-a840-4a8d-8b6a-78969f365f99-tls-certs\") pod \"590b41e5-a840-4a8d-8b6a-78969f365f99\" (UID: \"590b41e5-a840-4a8d-8b6a-78969f365f99\") "
Jul 09 18:12:41.248504 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.248466    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/590b41e5-a840-4a8d-8b6a-78969f365f99-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "590b41e5-a840-4a8d-8b6a-78969f365f99" (UID: "590b41e5-a840-4a8d-8b6a-78969f365f99"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:12:41.250163 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.250139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/590b41e5-a840-4a8d-8b6a-78969f365f99-kube-api-access-zgdkv" (OuterVolumeSpecName: "kube-api-access-zgdkv") pod "590b41e5-a840-4a8d-8b6a-78969f365f99" (UID: "590b41e5-a840-4a8d-8b6a-78969f365f99"). InnerVolumeSpecName "kube-api-access-zgdkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:12:41.250251 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.250143    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/590b41e5-a840-4a8d-8b6a-78969f365f99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "590b41e5-a840-4a8d-8b6a-78969f365f99" (UID: "590b41e5-a840-4a8d-8b6a-78969f365f99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:12:41.349641 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.349615    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/590b41e5-a840-4a8d-8b6a-78969f365f99-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:12:41.349641 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.349642    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgdkv\" (UniqueName: \"kubernetes.io/projected/590b41e5-a840-4a8d-8b6a-78969f365f99-kube-api-access-zgdkv\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:12:41.349803 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.349655    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/590b41e5-a840-4a8d-8b6a-78969f365f99-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:12:41.663224 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.663189    2577 generic.go:356] "Generic (PLEG): container finished" podID="590b41e5-a840-4a8d-8b6a-78969f365f99" containerID="17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448" exitCode=0
Jul 09 18:12:41.663405 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.663231    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"
Jul 09 18:12:41.663405 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.663273    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" event={"ID":"590b41e5-a840-4a8d-8b6a-78969f365f99","Type":"ContainerDied","Data":"17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448"}
Jul 09 18:12:41.663405 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.663320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl" event={"ID":"590b41e5-a840-4a8d-8b6a-78969f365f99","Type":"ContainerDied","Data":"8ef356b03128b320fcfa56af19710ef6bdfd46fdf3677e66ef4c90be27c14ada"}
Jul 09 18:12:41.663405 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.663341    2577 scope.go:117] "RemoveContainer" containerID="17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448"
Jul 09 18:12:41.672320 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.672303    2577 scope.go:117] "RemoveContainer" containerID="17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448"
Jul 09 18:12:41.672573 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:12:41.672556    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448\": container with ID starting with 17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448 not found: ID does not exist" containerID="17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448"
Jul 09 18:12:41.672620 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.672579    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448"} err="failed to get container status \"17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448\": rpc error: code = NotFound desc = could not find container \"17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448\": container with ID starting with 17fcca056882465e2f98ce51b040c8b23868ef1ce519a88ae32c7c9ec05e4448 not found: ID does not exist"
Jul 09 18:12:41.688860 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.686862    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"]
Jul 09 18:12:41.693138 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:41.693114    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-schett6vl"]
Jul 09 18:12:42.417226 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:42.417196    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="590b41e5-a840-4a8d-8b6a-78969f365f99" path="/var/lib/kubelet/pods/590b41e5-a840-4a8d-8b6a-78969f365f99/volumes"
Jul 09 18:12:54.084710 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.084675    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"]
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.084972    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="590b41e5-a840-4a8d-8b6a-78969f365f99" containerName="main"
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.084983    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="590b41e5-a840-4a8d-8b6a-78969f365f99" containerName="main"
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.084994    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerName="storage-initializer"
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.085000    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerName="storage-initializer"
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.085007    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerName="main"
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.085013    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerName="main"
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.085064    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="590b41e5-a840-4a8d-8b6a-78969f365f99" containerName="main"
Jul 09 18:12:54.085216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.085077    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96221ac9-3eab-4343-9023-26f7a8c1e875" containerName="main"
Jul 09 18:12:54.089323 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.089302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.091916 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.091893    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 09 18:12:54.092030 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.091966    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-gj7dj\""
Jul 09 18:12:54.099258 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.099228    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"]
Jul 09 18:12:54.148747 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.148720    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8lkn\" (UniqueName: \"kubernetes.io/projected/8061a397-457d-45a7-a61d-2c95d5f88f20-kube-api-access-b8lkn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.148915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.148767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8061a397-457d-45a7-a61d-2c95d5f88f20-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.148915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.148804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.148915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.148831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.250205 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.250170    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8lkn\" (UniqueName: \"kubernetes.io/projected/8061a397-457d-45a7-a61d-2c95d5f88f20-kube-api-access-b8lkn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.250355 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.250228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8061a397-457d-45a7-a61d-2c95d5f88f20-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.250355 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.250259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.250461 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.250429    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.250676 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.250654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-kserve-provision-location\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.250766 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.250750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-tokenizer-uds\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.252668 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.252647    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8061a397-457d-45a7-a61d-2c95d5f88f20-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.258789 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.258769    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8lkn\" (UniqueName: \"kubernetes.io/projected/8061a397-457d-45a7-a61d-2c95d5f88f20-kube-api-access-b8lkn\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") " pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.399445 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.399364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:54.530947 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.530918    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"]
Jul 09 18:12:54.533541 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:12:54.533510    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8061a397_457d_45a7_a61d_2c95d5f88f20.slice/crio-d10671f75eda168044c7c6b6e02aeb18123ff5ca50aba7479a23f42093c32748 WatchSource:0}: Error finding container d10671f75eda168044c7c6b6e02aeb18123ff5ca50aba7479a23f42093c32748: Status 404 returned error can't find the container with id d10671f75eda168044c7c6b6e02aeb18123ff5ca50aba7479a23f42093c32748
Jul 09 18:12:54.719295 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.719214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" event={"ID":"8061a397-457d-45a7-a61d-2c95d5f88f20","Type":"ContainerStarted","Data":"aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3"}
Jul 09 18:12:54.719295 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:54.719286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" event={"ID":"8061a397-457d-45a7-a61d-2c95d5f88f20","Type":"ContainerStarted","Data":"d10671f75eda168044c7c6b6e02aeb18123ff5ca50aba7479a23f42093c32748"}
Jul 09 18:12:55.724525 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:55.724496    2577 generic.go:356] "Generic (PLEG): container finished" podID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerID="aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3" exitCode=0
Jul 09 18:12:55.724911 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:55.724532    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" event={"ID":"8061a397-457d-45a7-a61d-2c95d5f88f20","Type":"ContainerDied","Data":"aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3"}
Jul 09 18:12:56.732995 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:56.732955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" event={"ID":"8061a397-457d-45a7-a61d-2c95d5f88f20","Type":"ContainerStarted","Data":"f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677"}
Jul 09 18:12:56.733613 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:56.733582    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:12:56.756192 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:12:56.756148    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" podStartSLOduration=2.0990582 podStartE2EDuration="2.756134574s" totalImagesPullingTime="0s" totalInitContainerRuntime="657.076374ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:56.754742382 +0000 UTC m=+2010.945102298" watchObservedRunningTime="2026-07-09 18:12:56.756134574 +0000 UTC m=+2010.946494506"
Jul 09 18:13:28.747374 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:13:28.747342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:14:26.353205 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:14:26.353172    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:14:26.353883 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:14:26.353866    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:17:01.001661 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.001566    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"]
Jul 09 18:17:01.010485 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:17:01.010446    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:17:01.010485 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:17:01.010485 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:17:01.010485 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677" cmd=["/bin/sleep","15"]
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.010494    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" message="" err=<
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:17:01.010508    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerName="main" containerID="cri-o://f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677"
Jul 09 18:17:01.010784 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.010545    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerName="main" containerID="cri-o://f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677" gracePeriod=60
Jul 09 18:17:01.267791 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.267730    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:17:01.412473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.412443    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8lkn\" (UniqueName: \"kubernetes.io/projected/8061a397-457d-45a7-a61d-2c95d5f88f20-kube-api-access-b8lkn\") pod \"8061a397-457d-45a7-a61d-2c95d5f88f20\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") "
Jul 09 18:17:01.412653 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.412497    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-tokenizer-uds\") pod \"8061a397-457d-45a7-a61d-2c95d5f88f20\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") "
Jul 09 18:17:01.412653 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.412532    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8061a397-457d-45a7-a61d-2c95d5f88f20-tls-certs\") pod \"8061a397-457d-45a7-a61d-2c95d5f88f20\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") "
Jul 09 18:17:01.412653 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.412556    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-kserve-provision-location\") pod \"8061a397-457d-45a7-a61d-2c95d5f88f20\" (UID: \"8061a397-457d-45a7-a61d-2c95d5f88f20\") "
Jul 09 18:17:01.412831 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.412769    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8061a397-457d-45a7-a61d-2c95d5f88f20" (UID: "8061a397-457d-45a7-a61d-2c95d5f88f20"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:01.413187 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.413165    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8061a397-457d-45a7-a61d-2c95d5f88f20" (UID: "8061a397-457d-45a7-a61d-2c95d5f88f20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:17:01.414472 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.414449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8061a397-457d-45a7-a61d-2c95d5f88f20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8061a397-457d-45a7-a61d-2c95d5f88f20" (UID: "8061a397-457d-45a7-a61d-2c95d5f88f20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:17:01.414554 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.414497    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8061a397-457d-45a7-a61d-2c95d5f88f20-kube-api-access-b8lkn" (OuterVolumeSpecName: "kube-api-access-b8lkn") pod "8061a397-457d-45a7-a61d-2c95d5f88f20" (UID: "8061a397-457d-45a7-a61d-2c95d5f88f20"). InnerVolumeSpecName "kube-api-access-b8lkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:17:01.513570 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.513542    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:17:01.513570 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.513567    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8lkn\" (UniqueName: \"kubernetes.io/projected/8061a397-457d-45a7-a61d-2c95d5f88f20-kube-api-access-b8lkn\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:17:01.513761 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.513578    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8061a397-457d-45a7-a61d-2c95d5f88f20-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:17:01.513761 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.513587    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8061a397-457d-45a7-a61d-2c95d5f88f20-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:17:01.841815 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.841781    2577 generic.go:356] "Generic (PLEG): container finished" podID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerID="f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677" exitCode=0
Jul 09 18:17:01.841991 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.841827    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"
Jul 09 18:17:01.841991 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.841874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" event={"ID":"8061a397-457d-45a7-a61d-2c95d5f88f20","Type":"ContainerDied","Data":"f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677"}
Jul 09 18:17:01.841991 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.841930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528" event={"ID":"8061a397-457d-45a7-a61d-2c95d5f88f20","Type":"ContainerDied","Data":"d10671f75eda168044c7c6b6e02aeb18123ff5ca50aba7479a23f42093c32748"}
Jul 09 18:17:01.841991 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.841959    2577 scope.go:117] "RemoveContainer" containerID="f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677"
Jul 09 18:17:01.853831 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.853767    2577 scope.go:117] "RemoveContainer" containerID="aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3"
Jul 09 18:17:01.861228 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.861209    2577 scope.go:117] "RemoveContainer" containerID="f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677"
Jul 09 18:17:01.861468 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:17:01.861446    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677\": container with ID starting with f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677 not found: ID does not exist" containerID="f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677"
Jul 09 18:17:01.861589 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.861472    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677"} err="failed to get container status \"f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677\": rpc error: code = NotFound desc = could not find container \"f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677\": container with ID starting with f8da43c09e839f47b223d209779e8de481f0ab859f4099d65d3713291768a677 not found: ID does not exist"
Jul 09 18:17:01.861589 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.861490    2577 scope.go:117] "RemoveContainer" containerID="aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3"
Jul 09 18:17:01.861735 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:17:01.861718    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3\": container with ID starting with aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3 not found: ID does not exist" containerID="aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3"
Jul 09 18:17:01.861804 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.861739    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3"} err="failed to get container status \"aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3\": rpc error: code = NotFound desc = could not find container \"aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3\": container with ID starting with aa67b9cf42519c9cc380c2df41a7734cedd03f1581ac8fd8251f3c744635bdd3 not found: ID does not exist"
Jul 09 18:17:01.881929 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.881879    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"]
Jul 09 18:17:01.889590 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:01.889564    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheh4528"]
Jul 09 18:17:02.418236 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:02.418194    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" path="/var/lib/kubelet/pods/8061a397-457d-45a7-a61d-2c95d5f88f20/volumes"
Jul 09 18:17:30.971628 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.971598    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"]
Jul 09 18:17:30.972017 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.971894    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerName="storage-initializer"
Jul 09 18:17:30.972017 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.971907    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerName="storage-initializer"
Jul 09 18:17:30.972017 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.971920    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerName="main"
Jul 09 18:17:30.972017 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.971926    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerName="main"
Jul 09 18:17:30.972017 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.971998    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8061a397-457d-45a7-a61d-2c95d5f88f20" containerName="main"
Jul 09 18:17:30.976280 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.976261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:30.979157 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.979132    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 18:17:30.979259 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.979244    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-k79g9\""
Jul 09 18:17:30.985345 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:30.985326    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"]
Jul 09 18:17:31.051150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.051123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzpl\" (UniqueName: \"kubernetes.io/projected/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kube-api-access-jgzpl\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.051296 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.051157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.051296 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.051180    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.051586 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.051298    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.151664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.151643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzpl\" (UniqueName: \"kubernetes.io/projected/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kube-api-access-jgzpl\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.151765 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.151671    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.151765 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.151698    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.151765 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.151723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.152096 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.152074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tokenizer-uds\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.152096 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.152087    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.154175 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.154149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.161182 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.161155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzpl\" (UniqueName: \"kubernetes.io/projected/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kube-api-access-jgzpl\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.286448 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.286411    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:31.430352 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.430324    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"]
Jul 09 18:17:31.433265 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:17:31.433236    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf57412e_6bf0_4fa8_ab4a_3c8ba0c199ad.slice/crio-198ea9e3f7e97515b0f00dd1acbdf50881e3a14d1a4d280d72fef93f2d48838e WatchSource:0}: Error finding container 198ea9e3f7e97515b0f00dd1acbdf50881e3a14d1a4d280d72fef93f2d48838e: Status 404 returned error can't find the container with id 198ea9e3f7e97515b0f00dd1acbdf50881e3a14d1a4d280d72fef93f2d48838e
Jul 09 18:17:31.977158 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.977115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" event={"ID":"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad","Type":"ContainerStarted","Data":"14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c"}
Jul 09 18:17:31.977731 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:31.977201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" event={"ID":"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad","Type":"ContainerStarted","Data":"198ea9e3f7e97515b0f00dd1acbdf50881e3a14d1a4d280d72fef93f2d48838e"}
Jul 09 18:17:32.982493 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:32.982451    2577 generic.go:356] "Generic (PLEG): container finished" podID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerID="14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c" exitCode=0
Jul 09 18:17:32.982978 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:32.982494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" event={"ID":"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad","Type":"ContainerDied","Data":"14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c"}
Jul 09 18:17:33.988009 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:33.987973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" event={"ID":"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad","Type":"ContainerStarted","Data":"721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282"}
Jul 09 18:17:33.988388 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:33.988086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:17:34.009977 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:17:34.009937    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" podStartSLOduration=3.322282088 podStartE2EDuration="4.00992274s" totalImagesPullingTime="0s" totalInitContainerRuntime="687.640652ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:17:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:17:34.009888704 +0000 UTC m=+2288.200248620" watchObservedRunningTime="2026-07-09 18:17:34.00992274 +0000 UTC m=+2288.200282637"
Jul 09 18:18:04.993828 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:04.993799    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:18:37.107315 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.107217    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"]
Jul 09 18:18:37.110646 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.110622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.114089 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.114065    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 18:18:37.115305 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.115284    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-5pjqs\""
Jul 09 18:18:37.126113 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.126089    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"]
Jul 09 18:18:37.257455 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.257395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.257643 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.257453    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e96e805-883b-4ada-8eac-141617c4725d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.257643 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.257536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxkr\" (UniqueName: \"kubernetes.io/projected/8e96e805-883b-4ada-8eac-141617c4725d-kube-api-access-fwxkr\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.257643 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.257611    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.359002 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.358927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.359002 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.358983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.359229 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.359028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e96e805-883b-4ada-8eac-141617c4725d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.359229 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.359076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxkr\" (UniqueName: \"kubernetes.io/projected/8e96e805-883b-4ada-8eac-141617c4725d-kube-api-access-fwxkr\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.359345 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.359272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.359345 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.359331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.361642 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.361611    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e96e805-883b-4ada-8eac-141617c4725d-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.367599 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.367579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxkr\" (UniqueName: \"kubernetes.io/projected/8e96e805-883b-4ada-8eac-141617c4725d-kube-api-access-fwxkr\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.420564 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.420527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:37.571522 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:37.571453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"]
Jul 09 18:18:37.573713 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:18:37.573680    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e96e805_883b_4ada_8eac_141617c4725d.slice/crio-a3368714a1191896e89c7fbe4dda5dd8fa8959ef51173759e67761461b121813 WatchSource:0}: Error finding container a3368714a1191896e89c7fbe4dda5dd8fa8959ef51173759e67761461b121813: Status 404 returned error can't find the container with id a3368714a1191896e89c7fbe4dda5dd8fa8959ef51173759e67761461b121813
Jul 09 18:18:38.285905 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:38.285868    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e96e805-883b-4ada-8eac-141617c4725d" containerID="4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8" exitCode=0
Jul 09 18:18:38.286264 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:38.285949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" event={"ID":"8e96e805-883b-4ada-8eac-141617c4725d","Type":"ContainerDied","Data":"4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8"}
Jul 09 18:18:38.286264 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:38.286011    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" event={"ID":"8e96e805-883b-4ada-8eac-141617c4725d","Type":"ContainerStarted","Data":"a3368714a1191896e89c7fbe4dda5dd8fa8959ef51173759e67761461b121813"}
Jul 09 18:18:39.301476 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:39.301427    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" event={"ID":"8e96e805-883b-4ada-8eac-141617c4725d","Type":"ContainerStarted","Data":"60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0"}
Jul 09 18:18:39.301898 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:39.301527    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:18:39.324494 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:18:39.323433    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" podStartSLOduration=1.6897296050000001 podStartE2EDuration="2.323415628s" totalImagesPullingTime="0s" totalInitContainerRuntime="633.686023ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:18:39.322767253 +0000 UTC m=+2353.513127169" watchObservedRunningTime="2026-07-09 18:18:39.323415628 +0000 UTC m=+2353.513775545"
Jul 09 18:19:10.307591 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:19:10.307563    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:19:26.384173 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:19:26.384143    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:19:26.385882 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:19:26.385852    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:20:30.518500 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:30.518467    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs: secret "custom-route-timeout-pd-test-kserve-self-signed-certs" not found
Jul 09 18:20:30.519468 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:30.518548    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs podName:cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad nodeName:}" failed. No retries permitted until 2026-07-09 18:20:31.018527082 +0000 UTC m=+2465.208887017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs") pod "custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" (UID: "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad") : secret "custom-route-timeout-pd-test-kserve-self-signed-certs" not found
Jul 09 18:20:30.521248 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:30.521214    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"]
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:30.811538    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282" cmd=["/bin/sleep","15"]
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:30.811584    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" message="" err=<
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:30.811595    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerName="main" containerID="cri-o://721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282"
Jul 09 18:20:30.811631 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:30.811628    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerName="main" containerID="cri-o://721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282" gracePeriod=60
Jul 09 18:20:31.021653 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:31.021627    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-self-signed-certs: secret "custom-route-timeout-pd-test-kserve-self-signed-certs" not found
Jul 09 18:20:31.021775 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:31.021697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs podName:cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad nodeName:}" failed. No retries permitted until 2026-07-09 18:20:32.021679409 +0000 UTC m=+2466.212039305 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs") pod "custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" (UID: "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad") : secret "custom-route-timeout-pd-test-kserve-self-signed-certs" not found
Jul 09 18:20:31.056523 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.056502    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:20:31.122030 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.121973    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tokenizer-uds\") pod \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") "
Jul 09 18:20:31.122135 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.122055    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kserve-provision-location\") pod \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") "
Jul 09 18:20:31.122135 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.122112    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs\") pod \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") "
Jul 09 18:20:31.122216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.122167    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgzpl\" (UniqueName: \"kubernetes.io/projected/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kube-api-access-jgzpl\") pod \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\" (UID: \"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad\") "
Jul 09 18:20:31.122264 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.122207    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" (UID: "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:31.122341 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.122329    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:20:31.122665 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.122643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" (UID: "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:20:31.124059 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.124031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kube-api-access-jgzpl" (OuterVolumeSpecName: "kube-api-access-jgzpl") pod "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" (UID: "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad"). InnerVolumeSpecName "kube-api-access-jgzpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:20:31.124149 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.124130    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" (UID: "cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:20:31.223150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.223101    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzpl\" (UniqueName: \"kubernetes.io/projected/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kube-api-access-jgzpl\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:20:31.223150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.223147    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:20:31.223150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.223158    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:20:31.808475 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.808445    2577 generic.go:356] "Generic (PLEG): container finished" podID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerID="721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282" exitCode=0
Jul 09 18:20:31.808874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.808495    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"
Jul 09 18:20:31.808874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.808529    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" event={"ID":"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad","Type":"ContainerDied","Data":"721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282"}
Jul 09 18:20:31.808874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.808586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6" event={"ID":"cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad","Type":"ContainerDied","Data":"198ea9e3f7e97515b0f00dd1acbdf50881e3a14d1a4d280d72fef93f2d48838e"}
Jul 09 18:20:31.808874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.808615    2577 scope.go:117] "RemoveContainer" containerID="721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282"
Jul 09 18:20:31.821584 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.821452    2577 scope.go:117] "RemoveContainer" containerID="14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c"
Jul 09 18:20:31.828812 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.828795    2577 scope.go:117] "RemoveContainer" containerID="721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282"
Jul 09 18:20:31.829072 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:31.829051    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282\": container with ID starting with 721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282 not found: ID does not exist" containerID="721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282"
Jul 09 18:20:31.829159 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.829075    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282"} err="failed to get container status \"721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282\": rpc error: code = NotFound desc = could not find container \"721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282\": container with ID starting with 721abc2c908e952e7fdbb1f0c008d4d3377816ea53c81ef10508054a8d108282 not found: ID does not exist"
Jul 09 18:20:31.829159 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.829093    2577 scope.go:117] "RemoveContainer" containerID="14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c"
Jul 09 18:20:31.829358 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:20:31.829340    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c\": container with ID starting with 14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c not found: ID does not exist" containerID="14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c"
Jul 09 18:20:31.829404 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.829364    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c"} err="failed to get container status \"14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c\": rpc error: code = NotFound desc = could not find container \"14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c\": container with ID starting with 14aaa94bea23529bb830871d36d328242112b86d4a621470d684a4a85d57b25c not found: ID does not exist"
Jul 09 18:20:31.836086 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.836065    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"]
Jul 09 18:20:31.840970 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:31.840943    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-router-scheduler-6c664d56p6"]
Jul 09 18:20:32.417059 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:20:32.417022    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" path="/var/lib/kubelet/pods/cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad/volumes"
Jul 09 18:21:29.770543 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.770511    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"]
Jul 09 18:21:29.771029 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.770802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerName="main"
Jul 09 18:21:29.771029 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.770813    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerName="main"
Jul 09 18:21:29.771029 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.770857    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerName="storage-initializer"
Jul 09 18:21:29.771029 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.770868    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerName="storage-initializer"
Jul 09 18:21:29.771029 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.770947    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf57412e-6bf0-4fa8-ab4a-3c8ba0c199ad" containerName="main"
Jul 09 18:21:29.774998 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.774977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.777731 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.777708    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-epp-sa-dockercfg-hvh5m\""
Jul 09 18:21:29.777883 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.777850    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 09 18:21:29.784674 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.784654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"]
Jul 09 18:21:29.871473 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.871450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.871597 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.871478    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lcwd\" (UniqueName: \"kubernetes.io/projected/40dd798c-6a26-41a5-a667-669522631f89-kube-api-access-4lcwd\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.871597 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.871507    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dd798c-6a26-41a5-a667-669522631f89-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.871597 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.871568    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.971918 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.971887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.972059 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.971922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lcwd\" (UniqueName: \"kubernetes.io/projected/40dd798c-6a26-41a5-a667-669522631f89-kube-api-access-4lcwd\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.972059 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.971962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dd798c-6a26-41a5-a667-669522631f89-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.972059 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.972017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.972339 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.972319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-tokenizer-uds\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.972339 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.972331    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.974481 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.974459    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dd798c-6a26-41a5-a667-669522631f89-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:29.981765 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:29.981745    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lcwd\" (UniqueName: \"kubernetes.io/projected/40dd798c-6a26-41a5-a667-669522631f89-kube-api-access-4lcwd\") pod \"router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:30.084637 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:30.084566    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:30.223920 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:30.223887    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"]
Jul 09 18:21:30.226672 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:21:30.226641    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dd798c_6a26_41a5_a667_669522631f89.slice/crio-964fa27e3bf1b69b24a89b3ebb9fdfb63893134584378c1e81246d8d56687df0 WatchSource:0}: Error finding container 964fa27e3bf1b69b24a89b3ebb9fdfb63893134584378c1e81246d8d56687df0: Status 404 returned error can't find the container with id 964fa27e3bf1b69b24a89b3ebb9fdfb63893134584378c1e81246d8d56687df0
Jul 09 18:21:31.081891 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:31.081861    2577 generic.go:356] "Generic (PLEG): container finished" podID="40dd798c-6a26-41a5-a667-669522631f89" containerID="b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6" exitCode=0
Jul 09 18:21:31.082254 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:31.081937    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" event={"ID":"40dd798c-6a26-41a5-a667-669522631f89","Type":"ContainerDied","Data":"b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6"}
Jul 09 18:21:31.082254 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:31.081981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" event={"ID":"40dd798c-6a26-41a5-a667-669522631f89","Type":"ContainerStarted","Data":"964fa27e3bf1b69b24a89b3ebb9fdfb63893134584378c1e81246d8d56687df0"}
Jul 09 18:21:32.088499 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:32.088466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" event={"ID":"40dd798c-6a26-41a5-a667-669522631f89","Type":"ContainerStarted","Data":"afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c"}
Jul 09 18:21:32.089031 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:32.088595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:21:32.115605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:32.112425    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" podStartSLOduration=2.482370857 podStartE2EDuration="3.112403999s" totalImagesPullingTime="0s" totalInitContainerRuntime="630.033142ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:21:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:21:32.108378846 +0000 UTC m=+2526.298738756" watchObservedRunningTime="2026-07-09 18:21:32.112403999 +0000 UTC m=+2526.302763918"
Jul 09 18:21:43.354871 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.354818    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"]
Jul 09 18:21:43.364176 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:21:43.364143    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:21:43.364176 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:21:43.364176 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:21:43.364176 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0" cmd=["/bin/sleep","15"]
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.364189    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" message="" err=<
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:21:43.364200    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" podUID="8e96e805-883b-4ada-8eac-141617c4725d" containerName="main" containerID="cri-o://60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0"
Jul 09 18:21:43.364498 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.364258    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" podUID="8e96e805-883b-4ada-8eac-141617c4725d" containerName="main" containerID="cri-o://60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0" gracePeriod=60
Jul 09 18:21:43.626268 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.626243    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:21:43.676257 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.676228    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwxkr\" (UniqueName: \"kubernetes.io/projected/8e96e805-883b-4ada-8eac-141617c4725d-kube-api-access-fwxkr\") pod \"8e96e805-883b-4ada-8eac-141617c4725d\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") "
Jul 09 18:21:43.676410 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.676262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e96e805-883b-4ada-8eac-141617c4725d-tls-certs\") pod \"8e96e805-883b-4ada-8eac-141617c4725d\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") "
Jul 09 18:21:43.676410 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.676305    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-tokenizer-uds\") pod \"8e96e805-883b-4ada-8eac-141617c4725d\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") "
Jul 09 18:21:43.676535 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.676419    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-kserve-provision-location\") pod \"8e96e805-883b-4ada-8eac-141617c4725d\" (UID: \"8e96e805-883b-4ada-8eac-141617c4725d\") "
Jul 09 18:21:43.676596 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.676536    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8e96e805-883b-4ada-8eac-141617c4725d" (UID: "8e96e805-883b-4ada-8eac-141617c4725d"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:43.676664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.676653    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:21:43.677190 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.677163    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e96e805-883b-4ada-8eac-141617c4725d" (UID: "8e96e805-883b-4ada-8eac-141617c4725d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:21:43.678857 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.678811    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e96e805-883b-4ada-8eac-141617c4725d-kube-api-access-fwxkr" (OuterVolumeSpecName: "kube-api-access-fwxkr") pod "8e96e805-883b-4ada-8eac-141617c4725d" (UID: "8e96e805-883b-4ada-8eac-141617c4725d"). InnerVolumeSpecName "kube-api-access-fwxkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:21:43.678946 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.678862    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e96e805-883b-4ada-8eac-141617c4725d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e96e805-883b-4ada-8eac-141617c4725d" (UID: "8e96e805-883b-4ada-8eac-141617c4725d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:21:43.777628 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.777608    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e96e805-883b-4ada-8eac-141617c4725d-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:21:43.777628 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.777630    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxkr\" (UniqueName: \"kubernetes.io/projected/8e96e805-883b-4ada-8eac-141617c4725d-kube-api-access-fwxkr\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:21:43.777752 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:43.777640    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e96e805-883b-4ada-8eac-141617c4725d-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:21:44.141231 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.141201    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e96e805-883b-4ada-8eac-141617c4725d" containerID="60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0" exitCode=0
Jul 09 18:21:44.141356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.141254    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"
Jul 09 18:21:44.141356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.141285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" event={"ID":"8e96e805-883b-4ada-8eac-141617c4725d","Type":"ContainerDied","Data":"60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0"}
Jul 09 18:21:44.141356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.141340    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4" event={"ID":"8e96e805-883b-4ada-8eac-141617c4725d","Type":"ContainerDied","Data":"a3368714a1191896e89c7fbe4dda5dd8fa8959ef51173759e67761461b121813"}
Jul 09 18:21:44.141460 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.141361    2577 scope.go:117] "RemoveContainer" containerID="60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0"
Jul 09 18:21:44.150621 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.150606    2577 scope.go:117] "RemoveContainer" containerID="4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8"
Jul 09 18:21:44.157696 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.157682    2577 scope.go:117] "RemoveContainer" containerID="60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0"
Jul 09 18:21:44.157930 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:21:44.157911    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0\": container with ID starting with 60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0 not found: ID does not exist" containerID="60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0"
Jul 09 18:21:44.158016 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.157938    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0"} err="failed to get container status \"60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0\": rpc error: code = NotFound desc = could not find container \"60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0\": container with ID starting with 60c051cb8c49a5b9c5c074813cd2988a9a934d77eea2e07272bf2820ff81def0 not found: ID does not exist"
Jul 09 18:21:44.158016 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.157954    2577 scope.go:117] "RemoveContainer" containerID="4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8"
Jul 09 18:21:44.158201 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:21:44.158185    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8\": container with ID starting with 4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8 not found: ID does not exist" containerID="4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8"
Jul 09 18:21:44.158242 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.158207    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8"} err="failed to get container status \"4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8\": rpc error: code = NotFound desc = could not find container \"4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8\": container with ID starting with 4505a865a41e12dd495a28be1b95d54acb5f7d374eed1fd310e08705dcdd3cd8 not found: ID does not exist"
Jul 09 18:21:44.163164 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.163144    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"]
Jul 09 18:21:44.166920 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.166899    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-2fnp4"]
Jul 09 18:21:44.422894 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:21:44.422794    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e96e805-883b-4ada-8eac-141617c4725d" path="/var/lib/kubelet/pods/8e96e805-883b-4ada-8eac-141617c4725d/volumes"
Jul 09 18:22:03.094071 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:03.094041    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:22:37.352313 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.352278    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"]
Jul 09 18:22:37.352952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.352569    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e96e805-883b-4ada-8eac-141617c4725d" containerName="storage-initializer"
Jul 09 18:22:37.352952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.352581    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e96e805-883b-4ada-8eac-141617c4725d" containerName="storage-initializer"
Jul 09 18:22:37.352952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.352594    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e96e805-883b-4ada-8eac-141617c4725d" containerName="main"
Jul 09 18:22:37.352952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.352599    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e96e805-883b-4ada-8eac-141617c4725d" containerName="main"
Jul 09 18:22:37.352952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.352652    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e96e805-883b-4ada-8eac-141617c4725d" containerName="main"
Jul 09 18:22:37.355754 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.355736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.358554 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.358533    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-zv7jd\""
Jul 09 18:22:37.358675 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.358573    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 18:22:37.367682 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.367658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"]
Jul 09 18:22:37.473280 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.473236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfrk9\" (UniqueName: \"kubernetes.io/projected/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kube-api-access-cfrk9\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.473280 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.473284    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.473497 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.473341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.473497 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.473386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.574612 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.574583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfrk9\" (UniqueName: \"kubernetes.io/projected/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kube-api-access-cfrk9\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.574746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.574616    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.574746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.574647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.574746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.574681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.574995 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.574976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.575053 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.575034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.577108 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.577090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.584411 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.584385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfrk9\" (UniqueName: \"kubernetes.io/projected/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kube-api-access-cfrk9\") pod \"stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.665618 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.665558    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:37.799102 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:37.799076    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"]
Jul 09 18:22:37.801356 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:22:37.801328    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174d8ba6_15b5_4a51_ad0f_8c7da918345c.slice/crio-6ebf49028186787af084fbf14ba3772766c070c09ddbb7829246849f118d2212 WatchSource:0}: Error finding container 6ebf49028186787af084fbf14ba3772766c070c09ddbb7829246849f118d2212: Status 404 returned error can't find the container with id 6ebf49028186787af084fbf14ba3772766c070c09ddbb7829246849f118d2212
Jul 09 18:22:38.387323 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:38.387290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" event={"ID":"174d8ba6-15b5-4a51-ad0f-8c7da918345c","Type":"ContainerStarted","Data":"b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92"}
Jul 09 18:22:38.387688 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:38.387350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" event={"ID":"174d8ba6-15b5-4a51-ad0f-8c7da918345c","Type":"ContainerStarted","Data":"6ebf49028186787af084fbf14ba3772766c070c09ddbb7829246849f118d2212"}
Jul 09 18:22:39.393464 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:39.393432    2577 generic.go:356] "Generic (PLEG): container finished" podID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerID="b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92" exitCode=0
Jul 09 18:22:39.393924 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:39.393502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" event={"ID":"174d8ba6-15b5-4a51-ad0f-8c7da918345c","Type":"ContainerDied","Data":"b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92"}
Jul 09 18:22:40.399851 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:40.399795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" event={"ID":"174d8ba6-15b5-4a51-ad0f-8c7da918345c","Type":"ContainerStarted","Data":"eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8"}
Jul 09 18:22:40.400225 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:40.399927    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:22:40.420987 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:22:40.420947    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" podStartSLOduration=2.7635239609999998 podStartE2EDuration="3.4209326s" totalImagesPullingTime="0s" totalInitContainerRuntime="657.408639ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:22:40.419285247 +0000 UTC m=+2594.609645160" watchObservedRunningTime="2026-07-09 18:22:40.4209326 +0000 UTC m=+2594.611292514"
Jul 09 18:23:11.406681 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:23:11.406651    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:24:26.410759 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:26.410677    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:24:26.415630 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:26.415607    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:24:34.567187 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.567156    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"]
Jul 09 18:24:34.576094 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:24:34.576054    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:24:34.576094 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:34.576094 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:34.576094 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c" cmd=["/bin/sleep","15"]
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.576110    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" message="" err=<
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:24:34.576124    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" podUID="40dd798c-6a26-41a5-a667-669522631f89" containerName="main" containerID="cri-o://afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c"
Jul 09 18:24:34.576323 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.576175    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" podUID="40dd798c-6a26-41a5-a667-669522631f89" containerName="main" containerID="cri-o://afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c" gracePeriod=60
Jul 09 18:24:34.823479 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.823424    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:24:34.912404 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.912365    2577 generic.go:356] "Generic (PLEG): container finished" podID="40dd798c-6a26-41a5-a667-669522631f89" containerID="afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c" exitCode=0
Jul 09 18:24:34.912592 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.912421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" event={"ID":"40dd798c-6a26-41a5-a667-669522631f89","Type":"ContainerDied","Data":"afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c"}
Jul 09 18:24:34.912592 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.912427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"
Jul 09 18:24:34.912592 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.912476    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng" event={"ID":"40dd798c-6a26-41a5-a667-669522631f89","Type":"ContainerDied","Data":"964fa27e3bf1b69b24a89b3ebb9fdfb63893134584378c1e81246d8d56687df0"}
Jul 09 18:24:34.912592 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.912510    2577 scope.go:117] "RemoveContainer" containerID="afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c"
Jul 09 18:24:34.922613 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.922589    2577 scope.go:117] "RemoveContainer" containerID="b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6"
Jul 09 18:24:34.932794 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.932772    2577 scope.go:117] "RemoveContainer" containerID="afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c"
Jul 09 18:24:34.933144 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:24:34.933123    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c\": container with ID starting with afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c not found: ID does not exist" containerID="afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c"
Jul 09 18:24:34.933235 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.933155    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c"} err="failed to get container status \"afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c\": rpc error: code = NotFound desc = could not find container \"afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c\": container with ID starting with afeed8ee95ddda39facbf5bee8e3f8c89d259c4ff8eb344d823d5656ca89914c not found: ID does not exist"
Jul 09 18:24:34.933235 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.933179    2577 scope.go:117] "RemoveContainer" containerID="b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6"
Jul 09 18:24:34.933506 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:24:34.933485    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6\": container with ID starting with b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6 not found: ID does not exist" containerID="b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6"
Jul 09 18:24:34.933628 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.933511    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6"} err="failed to get container status \"b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6\": rpc error: code = NotFound desc = could not find container \"b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6\": container with ID starting with b339a68acc6352c0fee82c84d9510548d7bd57c1a284e00bdeed8f9ae570b0b6 not found: ID does not exist"
Jul 09 18:24:34.990363 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.990339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-tokenizer-uds\") pod \"40dd798c-6a26-41a5-a667-669522631f89\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") "
Jul 09 18:24:34.990461 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.990375    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-kserve-provision-location\") pod \"40dd798c-6a26-41a5-a667-669522631f89\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") "
Jul 09 18:24:34.990461 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.990405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lcwd\" (UniqueName: \"kubernetes.io/projected/40dd798c-6a26-41a5-a667-669522631f89-kube-api-access-4lcwd\") pod \"40dd798c-6a26-41a5-a667-669522631f89\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") "
Jul 09 18:24:34.990461 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.990440    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dd798c-6a26-41a5-a667-669522631f89-tls-certs\") pod \"40dd798c-6a26-41a5-a667-669522631f89\" (UID: \"40dd798c-6a26-41a5-a667-669522631f89\") "
Jul 09 18:24:34.990623 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.990598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "40dd798c-6a26-41a5-a667-669522631f89" (UID: "40dd798c-6a26-41a5-a667-669522631f89"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:34.991033 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.991012    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40dd798c-6a26-41a5-a667-669522631f89" (UID: "40dd798c-6a26-41a5-a667-669522631f89"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:24:34.992367 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.992345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40dd798c-6a26-41a5-a667-669522631f89-kube-api-access-4lcwd" (OuterVolumeSpecName: "kube-api-access-4lcwd") pod "40dd798c-6a26-41a5-a667-669522631f89" (UID: "40dd798c-6a26-41a5-a667-669522631f89"). InnerVolumeSpecName "kube-api-access-4lcwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:24:34.992446 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:34.992392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40dd798c-6a26-41a5-a667-669522631f89-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40dd798c-6a26-41a5-a667-669522631f89" (UID: "40dd798c-6a26-41a5-a667-669522631f89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:24:35.091587 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:35.091534    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:24:35.091587 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:35.091557    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40dd798c-6a26-41a5-a667-669522631f89-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:24:35.091587 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:35.091567    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lcwd\" (UniqueName: \"kubernetes.io/projected/40dd798c-6a26-41a5-a667-669522631f89-kube-api-access-4lcwd\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:24:35.091587 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:35.091575    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40dd798c-6a26-41a5-a667-669522631f89-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:24:35.255354 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:35.255329    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"]
Jul 09 18:24:35.264373 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:35.260109    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-router-scheduler-549db8f6f92qng"]
Jul 09 18:24:36.419829 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:24:36.419785    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40dd798c-6a26-41a5-a667-669522631f89" path="/var/lib/kubelet/pods/40dd798c-6a26-41a5-a667-669522631f89/volumes"
Jul 09 18:25:27.396590 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.396553    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"]
Jul 09 18:25:27.405079 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:25:27.405046    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:25:27.405079 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:25:27.405079 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:25:27.405079 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8" cmd=["/bin/sleep","15"]
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.405094    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" message="" err=<
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:25:27.405106    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerName="main" containerID="cri-o://eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8"
Jul 09 18:25:27.405346 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.405150    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerName="main" containerID="cri-o://eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8" gracePeriod=60
Jul 09 18:25:27.649649 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.649597    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:25:27.683606 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.683580    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfrk9\" (UniqueName: \"kubernetes.io/projected/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kube-api-access-cfrk9\") pod \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") "
Jul 09 18:25:27.683763 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.683646    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kserve-provision-location\") pod \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") "
Jul 09 18:25:27.684245 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.684220    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "174d8ba6-15b5-4a51-ad0f-8c7da918345c" (UID: "174d8ba6-15b5-4a51-ad0f-8c7da918345c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:27.685688 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.685659    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kube-api-access-cfrk9" (OuterVolumeSpecName: "kube-api-access-cfrk9") pod "174d8ba6-15b5-4a51-ad0f-8c7da918345c" (UID: "174d8ba6-15b5-4a51-ad0f-8c7da918345c"). InnerVolumeSpecName "kube-api-access-cfrk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:25:27.784380 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.784351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tokenizer-uds\") pod \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") "
Jul 09 18:25:27.784539 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.784394    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tls-certs\") pod \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\" (UID: \"174d8ba6-15b5-4a51-ad0f-8c7da918345c\") "
Jul 09 18:25:27.784612 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.784552    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:25:27.784612 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.784570    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfrk9\" (UniqueName: \"kubernetes.io/projected/174d8ba6-15b5-4a51-ad0f-8c7da918345c-kube-api-access-cfrk9\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:25:27.784728 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.784621    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "174d8ba6-15b5-4a51-ad0f-8c7da918345c" (UID: "174d8ba6-15b5-4a51-ad0f-8c7da918345c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:25:27.786483 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.786454    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "174d8ba6-15b5-4a51-ad0f-8c7da918345c" (UID: "174d8ba6-15b5-4a51-ad0f-8c7da918345c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:25:27.885847 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.885812    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:25:27.885847 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:27.885851    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/174d8ba6-15b5-4a51-ad0f-8c7da918345c-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:25:28.168167 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.168142    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" event={"ID":"174d8ba6-15b5-4a51-ad0f-8c7da918345c","Type":"ContainerDied","Data":"eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8"}
Jul 09 18:25:28.168306 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.168177    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"
Jul 09 18:25:28.168306 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.168214    2577 scope.go:117] "RemoveContainer" containerID="eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8"
Jul 09 18:25:28.168306 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.167981    2577 generic.go:356] "Generic (PLEG): container finished" podID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerID="eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8" exitCode=0
Jul 09 18:25:28.168461 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.168311    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2" event={"ID":"174d8ba6-15b5-4a51-ad0f-8c7da918345c","Type":"ContainerDied","Data":"6ebf49028186787af084fbf14ba3772766c070c09ddbb7829246849f118d2212"}
Jul 09 18:25:28.176686 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.176673    2577 scope.go:117] "RemoveContainer" containerID="b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92"
Jul 09 18:25:28.183964 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.183889    2577 scope.go:117] "RemoveContainer" containerID="eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8"
Jul 09 18:25:28.184126 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:25:28.184109    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8\": container with ID starting with eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8 not found: ID does not exist" containerID="eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8"
Jul 09 18:25:28.184172 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.184132    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8"} err="failed to get container status \"eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8\": rpc error: code = NotFound desc = could not find container \"eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8\": container with ID starting with eee1459623ba3396f335cadefb7cb6d5a1024357b422f0de232398c55fcbd0a8 not found: ID does not exist"
Jul 09 18:25:28.184172 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.184147    2577 scope.go:117] "RemoveContainer" containerID="b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92"
Jul 09 18:25:28.184400 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:25:28.184382    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92\": container with ID starting with b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92 not found: ID does not exist" containerID="b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92"
Jul 09 18:25:28.184453 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.184404    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92"} err="failed to get container status \"b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92\": rpc error: code = NotFound desc = could not find container \"b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92\": container with ID starting with b8ca302532f49ad62848a2ee854ab7d825ed6e0dbfb8cd6a6f6de6aba0e53e92 not found: ID does not exist"
Jul 09 18:25:28.192596 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.192577    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"]
Jul 09 18:25:28.196044 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.196026    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5c8954f69-mh7p2"]
Jul 09 18:25:28.417803 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:25:28.417767    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" path="/var/lib/kubelet/pods/174d8ba6-15b5-4a51-ad0f-8c7da918345c/volumes"
Jul 09 18:27:40.845011 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.844973    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845373    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerName="storage-initializer"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845390    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerName="storage-initializer"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845412    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerName="main"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845423    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerName="main"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845443    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40dd798c-6a26-41a5-a667-669522631f89" containerName="main"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845451    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="40dd798c-6a26-41a5-a667-669522631f89" containerName="main"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845466    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40dd798c-6a26-41a5-a667-669522631f89" containerName="storage-initializer"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845474    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="40dd798c-6a26-41a5-a667-669522631f89" containerName="storage-initializer"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845551    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="40dd798c-6a26-41a5-a667-669522631f89" containerName="main"
Jul 09 18:27:40.845605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.845564    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="174d8ba6-15b5-4a51-ad0f-8c7da918345c" containerName="main"
Jul 09 18:27:40.854187 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.854144    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.857282 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.857253    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 09 18:27:40.857431 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.857379    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-5w6nc\""
Jul 09 18:27:40.861086 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.861058    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:27:40.872570 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.872540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.872681 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.872633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnkbh\" (UniqueName: \"kubernetes.io/projected/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kube-api-access-jnkbh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.872681 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.872670    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.872781 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.872694    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.872781 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.872726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.872781 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.872749    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.872918 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.872793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974131 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnkbh\" (UniqueName: \"kubernetes.io/projected/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kube-api-access-jnkbh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974643 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974643 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974619    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974724 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974724 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.974724 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.974713    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.976356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.976337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.976696 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.976680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:40.984832 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:40.984808    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnkbh\" (UniqueName: \"kubernetes.io/projected/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kube-api-access-jnkbh\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:41.019720 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.019688    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"]
Jul 09 18:27:41.023256 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.023238    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.026664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.026636    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-jrl8s\""
Jul 09 18:27:41.036736 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.036714    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"]
Jul 09 18:27:41.074737 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.074715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll8d5\" (UniqueName: \"kubernetes.io/projected/260933b2-f00e-4b04-ae7f-2b023cb751ba-kube-api-access-ll8d5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.074881 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.074757    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260933b2-f00e-4b04-ae7f-2b023cb751ba-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.074881 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.074781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.074881 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.074806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.167128 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.167035    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:27:41.175415 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.175384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll8d5\" (UniqueName: \"kubernetes.io/projected/260933b2-f00e-4b04-ae7f-2b023cb751ba-kube-api-access-ll8d5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.175514 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.175427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260933b2-f00e-4b04-ae7f-2b023cb751ba-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.175514 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.175452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.175514 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.175481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.175911 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.175874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-tokenizer-uds\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.176016 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.175921    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-kserve-provision-location\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.178483 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.178455    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260933b2-f00e-4b04-ae7f-2b023cb751ba-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.184706 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.184676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll8d5\" (UniqueName: \"kubernetes.io/projected/260933b2-f00e-4b04-ae7f-2b023cb751ba-kube-api-access-ll8d5\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") " pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.311139 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.311094    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:27:41.315713 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:27:41.315684    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15b57d9_91d8_490e_9a6b_7a0de18bbcfb.slice/crio-f4905d5a42600681aa6cf67e1b396f9b751ca12eaad1b3f895ec5b37f220d708 WatchSource:0}: Error finding container f4905d5a42600681aa6cf67e1b396f9b751ca12eaad1b3f895ec5b37f220d708: Status 404 returned error can't find the container with id f4905d5a42600681aa6cf67e1b396f9b751ca12eaad1b3f895ec5b37f220d708
Jul 09 18:27:41.333161 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.333049    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:41.475715 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.475679    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"]
Jul 09 18:27:41.479216 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:27:41.479191    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260933b2_f00e_4b04_ae7f_2b023cb751ba.slice/crio-7ba470886d4b5534bc3143e54a054a65039855b616c94c4c0f8a0a354f3b2790 WatchSource:0}: Error finding container 7ba470886d4b5534bc3143e54a054a65039855b616c94c4c0f8a0a354f3b2790: Status 404 returned error can't find the container with id 7ba470886d4b5534bc3143e54a054a65039855b616c94c4c0f8a0a354f3b2790
Jul 09 18:27:41.744153 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.744043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" event={"ID":"260933b2-f00e-4b04-ae7f-2b023cb751ba","Type":"ContainerStarted","Data":"f2e0eddd58e208215f4a9bfcb304b9adf133f153a19511d2fd594765038a1f42"}
Jul 09 18:27:41.744153 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.744113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" event={"ID":"260933b2-f00e-4b04-ae7f-2b023cb751ba","Type":"ContainerStarted","Data":"7ba470886d4b5534bc3143e54a054a65039855b616c94c4c0f8a0a354f3b2790"}
Jul 09 18:27:41.745757 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.745722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb","Type":"ContainerStarted","Data":"5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f"}
Jul 09 18:27:41.745905 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:41.745777    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb","Type":"ContainerStarted","Data":"f4905d5a42600681aa6cf67e1b396f9b751ca12eaad1b3f895ec5b37f220d708"}
Jul 09 18:27:42.751968 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:42.751932    2577 generic.go:356] "Generic (PLEG): container finished" podID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerID="f2e0eddd58e208215f4a9bfcb304b9adf133f153a19511d2fd594765038a1f42" exitCode=0
Jul 09 18:27:42.752371 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:42.752024    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" event={"ID":"260933b2-f00e-4b04-ae7f-2b023cb751ba","Type":"ContainerDied","Data":"f2e0eddd58e208215f4a9bfcb304b9adf133f153a19511d2fd594765038a1f42"}
Jul 09 18:27:43.758094 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:43.758059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" event={"ID":"260933b2-f00e-4b04-ae7f-2b023cb751ba","Type":"ContainerStarted","Data":"994e7c72f062941227e438a592cb1d120470568d90ac87de5aa9140bda189706"}
Jul 09 18:27:43.758472 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:43.758192    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:27:43.783435 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:43.783388    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" podStartSLOduration=3.090701855 podStartE2EDuration="3.783375701s" totalImagesPullingTime="0s" totalInitContainerRuntime="692.673846ms" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:27:43.78199065 +0000 UTC m=+2897.972350565" watchObservedRunningTime="2026-07-09 18:27:43.783375701 +0000 UTC m=+2897.973735627"
Jul 09 18:27:45.769991 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:45.769956    2577 generic.go:356] "Generic (PLEG): container finished" podID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerID="5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f" exitCode=0
Jul 09 18:27:45.770466 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:45.770029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb","Type":"ContainerDied","Data":"5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f"}
Jul 09 18:27:45.771353 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:27:45.771338    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:28:14.768267 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:28:14.768240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:28:14.940982 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:28:14.940899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb","Type":"ContainerStarted","Data":"f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187"}
Jul 09 18:28:14.969033 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:28:14.968969    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.80549055 podStartE2EDuration="34.968947075s" totalImagesPullingTime="28.778988474s" totalInitContainerRuntime="3.384468051s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:28:14.966609266 +0000 UTC m=+2929.156969183" watchObservedRunningTime="2026-07-09 18:28:14.968947075 +0000 UTC m=+2929.159306991"
Jul 09 18:29:26.440173 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:29:26.440056    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:29:26.447050 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:29:26.447029    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:30:46.435047 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.434996    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"]
Jul 09 18:30:46.443510 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:30:46.443471    2577 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 18:30:46.443510 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:46.443510 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:46.443510 ip-10-0-141-44 kubenswrapper[2577]:  > containerID="994e7c72f062941227e438a592cb1d120470568d90ac87de5aa9140bda189706" cmd=["/bin/sleep","15"]
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.443530    2577 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" message="" err=<
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]:  >
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:30:46.443545    2577 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]:         , stdout: , stderr: , exit code -1
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]:  > pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerName="main" containerID="cri-o://994e7c72f062941227e438a592cb1d120470568d90ac87de5aa9140bda189706"
Jul 09 18:30:46.443746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.443599    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerName="main" containerID="cri-o://994e7c72f062941227e438a592cb1d120470568d90ac87de5aa9140bda189706" gracePeriod=60
Jul 09 18:30:46.669123 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.669087    2577 generic.go:356] "Generic (PLEG): container finished" podID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerID="994e7c72f062941227e438a592cb1d120470568d90ac87de5aa9140bda189706" exitCode=0
Jul 09 18:30:46.669265 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.669156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" event={"ID":"260933b2-f00e-4b04-ae7f-2b023cb751ba","Type":"ContainerDied","Data":"994e7c72f062941227e438a592cb1d120470568d90ac87de5aa9140bda189706"}
Jul 09 18:30:46.690633 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.690574    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:30:46.791214 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.791184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-tokenizer-uds\") pod \"260933b2-f00e-4b04-ae7f-2b023cb751ba\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") "
Jul 09 18:30:46.791415 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.791232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-kserve-provision-location\") pod \"260933b2-f00e-4b04-ae7f-2b023cb751ba\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") "
Jul 09 18:30:46.791415 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.791254    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ll8d5\" (UniqueName: \"kubernetes.io/projected/260933b2-f00e-4b04-ae7f-2b023cb751ba-kube-api-access-ll8d5\") pod \"260933b2-f00e-4b04-ae7f-2b023cb751ba\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") "
Jul 09 18:30:46.791415 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.791281    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260933b2-f00e-4b04-ae7f-2b023cb751ba-tls-certs\") pod \"260933b2-f00e-4b04-ae7f-2b023cb751ba\" (UID: \"260933b2-f00e-4b04-ae7f-2b023cb751ba\") "
Jul 09 18:30:46.791916 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.791889    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "260933b2-f00e-4b04-ae7f-2b023cb751ba" (UID: "260933b2-f00e-4b04-ae7f-2b023cb751ba"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:46.792230 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.792197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "260933b2-f00e-4b04-ae7f-2b023cb751ba" (UID: "260933b2-f00e-4b04-ae7f-2b023cb751ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:30:46.793592 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.793569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/260933b2-f00e-4b04-ae7f-2b023cb751ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "260933b2-f00e-4b04-ae7f-2b023cb751ba" (UID: "260933b2-f00e-4b04-ae7f-2b023cb751ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:30:46.793917 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.793887    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260933b2-f00e-4b04-ae7f-2b023cb751ba-kube-api-access-ll8d5" (OuterVolumeSpecName: "kube-api-access-ll8d5") pod "260933b2-f00e-4b04-ae7f-2b023cb751ba" (UID: "260933b2-f00e-4b04-ae7f-2b023cb751ba"). InnerVolumeSpecName "kube-api-access-ll8d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:30:46.892068 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.892045    2577 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-tokenizer-uds\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:30:46.892068 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.892067    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/260933b2-f00e-4b04-ae7f-2b023cb751ba-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:30:46.892209 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.892078    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll8d5\" (UniqueName: \"kubernetes.io/projected/260933b2-f00e-4b04-ae7f-2b023cb751ba-kube-api-access-ll8d5\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:30:46.892209 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:46.892088    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/260933b2-f00e-4b04-ae7f-2b023cb751ba-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:30:47.674694 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:47.674657    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8" event={"ID":"260933b2-f00e-4b04-ae7f-2b023cb751ba","Type":"ContainerDied","Data":"7ba470886d4b5534bc3143e54a054a65039855b616c94c4c0f8a0a354f3b2790"}
Jul 09 18:30:47.674694 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:47.674695    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"
Jul 09 18:30:47.675182 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:47.674722    2577 scope.go:117] "RemoveContainer" containerID="994e7c72f062941227e438a592cb1d120470568d90ac87de5aa9140bda189706"
Jul 09 18:30:47.684022 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:47.684003    2577 scope.go:117] "RemoveContainer" containerID="f2e0eddd58e208215f4a9bfcb304b9adf133f153a19511d2fd594765038a1f42"
Jul 09 18:30:47.696720 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:47.696700    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"]
Jul 09 18:30:47.702098 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:47.702077    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scherv8q8"]
Jul 09 18:30:48.417074 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:30:48.417042    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" path="/var/lib/kubelet/pods/260933b2-f00e-4b04-ae7f-2b023cb751ba/volumes"
Jul 09 18:31:03.191110 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:03.191077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:31:07.421581 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.421553    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"]
Jul 09 18:31:07.421954 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.421818    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerName="main"
Jul 09 18:31:07.421954 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.421828    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerName="main"
Jul 09 18:31:07.421954 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.421849    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerName="storage-initializer"
Jul 09 18:31:07.421954 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.421855    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerName="storage-initializer"
Jul 09 18:31:07.421954 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.421908    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="260933b2-f00e-4b04-ae7f-2b023cb751ba" containerName="main"
Jul 09 18:31:07.426138 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.426116    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.428988 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.428969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 09 18:31:07.438150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.438129    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"]
Jul 09 18:31:07.556168 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.556141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-tmp-dir\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.556280 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.556178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-home\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.556324 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.556279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c501599-ef57-475f-8228-bc61218f7e8a-tls-certs\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.556364 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.556332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-dshm\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.556399 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.556361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-kserve-provision-location\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.556399 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.556387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j27cm\" (UniqueName: \"kubernetes.io/projected/8c501599-ef57-475f-8228-bc61218f7e8a-kube-api-access-j27cm\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.556472 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.556440    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-model-cache\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657201 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c501599-ef57-475f-8228-bc61218f7e8a-tls-certs\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657386 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-dshm\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657386 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657256    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-kserve-provision-location\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657386 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j27cm\" (UniqueName: \"kubernetes.io/projected/8c501599-ef57-475f-8228-bc61218f7e8a-kube-api-access-j27cm\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657386 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-model-cache\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657619 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-tmp-dir\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657619 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-home\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.657929 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657895    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-home\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.658137 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.657950    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-kserve-provision-location\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.658137 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.658113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-model-cache\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.658137 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.658130    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-tmp-dir\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.660515 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.660484    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c501599-ef57-475f-8228-bc61218f7e8a-tls-certs\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.660678 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.660656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-dshm\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.670234 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.670205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j27cm\" (UniqueName: \"kubernetes.io/projected/8c501599-ef57-475f-8228-bc61218f7e8a-kube-api-access-j27cm\") pod \"tls-verification-test-kserve-6799cd4648-xlzj2\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.736029 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.735977    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:07.883271 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:07.883222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"]
Jul 09 18:31:07.884639 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:31:07.884607    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c501599_ef57_475f_8228_bc61218f7e8a.slice/crio-09dab482c1904f6bc771312bc97a40675fd90cda0eadfbee0102c8a253b92df5 WatchSource:0}: Error finding container 09dab482c1904f6bc771312bc97a40675fd90cda0eadfbee0102c8a253b92df5: Status 404 returned error can't find the container with id 09dab482c1904f6bc771312bc97a40675fd90cda0eadfbee0102c8a253b92df5
Jul 09 18:31:08.769998 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:08.769967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" event={"ID":"8c501599-ef57-475f-8228-bc61218f7e8a","Type":"ContainerStarted","Data":"218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9"}
Jul 09 18:31:08.770479 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:08.770018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" event={"ID":"8c501599-ef57-475f-8228-bc61218f7e8a","Type":"ContainerStarted","Data":"09dab482c1904f6bc771312bc97a40675fd90cda0eadfbee0102c8a253b92df5"}
Jul 09 18:31:11.785868 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:11.785822    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c501599-ef57-475f-8228-bc61218f7e8a" containerID="218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9" exitCode=0
Jul 09 18:31:11.786235 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:11.785879    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" event={"ID":"8c501599-ef57-475f-8228-bc61218f7e8a","Type":"ContainerDied","Data":"218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9"}
Jul 09 18:31:12.794086 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:12.794044    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" event={"ID":"8c501599-ef57-475f-8228-bc61218f7e8a","Type":"ContainerStarted","Data":"9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c"}
Jul 09 18:31:12.821179 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:12.821128    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podStartSLOduration=2.633808138 podStartE2EDuration="5.821098347s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.187290209s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:12.81837625 +0000 UTC m=+3107.008736165" watchObservedRunningTime="2026-07-09 18:31:12.821098347 +0000 UTC m=+3107.011458277"
Jul 09 18:31:13.736673 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:13.736629    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:17.736496 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:17.736448    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:31:17.737993 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:17.737958    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:31:18.213407 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:18.213371    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerName="main" containerID="cri-o://f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187" gracePeriod=45
Jul 09 18:31:18.984359 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:18.984327    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:31:19.055118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.054906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tls-certs\") pod \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") "
Jul 09 18:31:19.055311 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.055161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-dshm\") pod \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") "
Jul 09 18:31:19.055311 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.055192    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kserve-provision-location\") pod \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") "
Jul 09 18:31:19.055311 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.055252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnkbh\" (UniqueName: \"kubernetes.io/projected/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kube-api-access-jnkbh\") pod \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") "
Jul 09 18:31:19.055311 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.055302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-home\") pod \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") "
Jul 09 18:31:19.055557 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.055329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-model-cache\") pod \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") "
Jul 09 18:31:19.055557 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.055362    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tmp-dir\") pod \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\" (UID: \"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb\") "
Jul 09 18:31:19.057276 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.057230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" (UID: "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:19.057620 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.057591    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-dshm" (OuterVolumeSpecName: "dshm") pod "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" (UID: "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:19.057812 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.057772    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" (UID: "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:31:19.058072 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.058045    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kube-api-access-jnkbh" (OuterVolumeSpecName: "kube-api-access-jnkbh") pod "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" (UID: "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb"). InnerVolumeSpecName "kube-api-access-jnkbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:31:19.058361 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.058335    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-home" (OuterVolumeSpecName: "home") pod "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" (UID: "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:19.073140 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.073115    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" (UID: "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:19.120121 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.120084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" (UID: "a15b57d9-91d8-490e-9a6b-7a0de18bbcfb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:31:19.156950 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.156926    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:31:19.156950 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.156947    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:31:19.157118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.156957    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:31:19.157118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.156965    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:31:19.157118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.156972    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:31:19.157118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.156980    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:31:19.157118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.156989    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnkbh\" (UniqueName: \"kubernetes.io/projected/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb-kube-api-access-jnkbh\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:31:19.828641 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.828603    2577 generic.go:356] "Generic (PLEG): container finished" podID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerID="f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187" exitCode=0
Jul 09 18:31:19.829024 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.828656    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 18:31:19.829024 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.828675    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb","Type":"ContainerDied","Data":"f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187"}
Jul 09 18:31:19.829024 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.828722    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a15b57d9-91d8-490e-9a6b-7a0de18bbcfb","Type":"ContainerDied","Data":"f4905d5a42600681aa6cf67e1b396f9b751ca12eaad1b3f895ec5b37f220d708"}
Jul 09 18:31:19.829024 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.828747    2577 scope.go:117] "RemoveContainer" containerID="f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187"
Jul 09 18:31:19.838456 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.838440    2577 scope.go:117] "RemoveContainer" containerID="5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f"
Jul 09 18:31:19.847799 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.847775    2577 scope.go:117] "RemoveContainer" containerID="f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187"
Jul 09 18:31:19.848104 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:31:19.848082    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187\": container with ID starting with f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187 not found: ID does not exist" containerID="f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187"
Jul 09 18:31:19.848216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.848116    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187"} err="failed to get container status \"f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187\": rpc error: code = NotFound desc = could not find container \"f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187\": container with ID starting with f9e6bf3acb039c61dc27e9fa40e36980f799589d3ecdc1968cc76ffdc775e187 not found: ID does not exist"
Jul 09 18:31:19.848216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.848142    2577 scope.go:117] "RemoveContainer" containerID="5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f"
Jul 09 18:31:19.848401 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:31:19.848385    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f\": container with ID starting with 5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f not found: ID does not exist" containerID="5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f"
Jul 09 18:31:19.848457 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.848410    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f"} err="failed to get container status \"5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f\": rpc error: code = NotFound desc = could not find container \"5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f\": container with ID starting with 5a73c7e89f76d7b97f7a23e0d9774453806b21e600ef1f1e94f60378432cea3f not found: ID does not exist"
Jul 09 18:31:19.857660 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.857630    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:31:19.859604 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:19.859583    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 18:31:20.417583 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:20.417551    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" path="/var/lib/kubelet/pods/a15b57d9-91d8-490e-9a6b-7a0de18bbcfb/volumes"
Jul 09 18:31:27.736424 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:27.736373    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:31:30.700360 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.700327    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"]
Jul 09 18:31:30.700959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.700756    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerName="main"
Jul 09 18:31:30.700959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.700774    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerName="main"
Jul 09 18:31:30.700959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.700799    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerName="storage-initializer"
Jul 09 18:31:30.700959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.700808    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerName="storage-initializer"
Jul 09 18:31:30.700959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.700910    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a15b57d9-91d8-490e-9a6b-7a0de18bbcfb" containerName="main"
Jul 09 18:31:30.704096 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.704076    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.708515 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.708494    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 09 18:31:30.727254 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.727220    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"]
Jul 09 18:31:30.854610 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.854572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.854610 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.854609    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.854814 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.854635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.854814 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.854744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.854814 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.854785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.854961 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.854864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl27w\" (UniqueName: \"kubernetes.io/projected/8abdb8cd-36f1-42de-a65e-ed6af385fac7-kube-api-access-hl27w\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.955953 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.955876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.955953 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.955910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.955953 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.955945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl27w\" (UniqueName: \"kubernetes.io/projected/8abdb8cd-36f1-42de-a65e-ed6af385fac7-kube-api-access-hl27w\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.956212 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.955989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.956212 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.956017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.956212 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.956056    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.956423 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.956399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.956493 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.956467    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.956549 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.956514    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.958215 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.958195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.958335 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.958325    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:30.976216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:30.976193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl27w\" (UniqueName: \"kubernetes.io/projected/8abdb8cd-36f1-42de-a65e-ed6af385fac7-kube-api-access-hl27w\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:31.016351 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:31.016328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:31.161700 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:31.161677    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"]
Jul 09 18:31:31.164199 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:31:31.164169    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8abdb8cd_36f1_42de_a65e_ed6af385fac7.slice/crio-1931800e70f0794d435cb37e2a894cc4949de0786d40050a7c1a831ef18e574a WatchSource:0}: Error finding container 1931800e70f0794d435cb37e2a894cc4949de0786d40050a7c1a831ef18e574a: Status 404 returned error can't find the container with id 1931800e70f0794d435cb37e2a894cc4949de0786d40050a7c1a831ef18e574a
Jul 09 18:31:31.893395 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:31.893358    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw" event={"ID":"8abdb8cd-36f1-42de-a65e-ed6af385fac7","Type":"ContainerStarted","Data":"bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe"}
Jul 09 18:31:31.893902 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:31.893435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw" event={"ID":"8abdb8cd-36f1-42de-a65e-ed6af385fac7","Type":"ContainerStarted","Data":"1931800e70f0794d435cb37e2a894cc4949de0786d40050a7c1a831ef18e574a"}
Jul 09 18:31:31.916053 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:31.916006    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw" podStartSLOduration=1.915992058 podStartE2EDuration="1.915992058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:31:31.91512595 +0000 UTC m=+3126.105485866" watchObservedRunningTime="2026-07-09 18:31:31.915992058 +0000 UTC m=+3126.106351973"
Jul 09 18:31:32.016912 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:32.016882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:37.737020 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:37.736973    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:31:41.017209 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:41.017168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:41.029752 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:41.029722    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:41.954442 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:41.954414    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:31:47.736766 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:47.736726    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:31:57.737346 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:31:57.737252    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:32:05.264670 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:05.264634    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"]
Jul 09 18:32:07.737222 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:07.737178    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:32:17.736578 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:17.736532    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:32:20.290139 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.290094    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw" podUID="8abdb8cd-36f1-42de-a65e-ed6af385fac7" containerName="main" containerID="cri-o://bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe" gracePeriod=45
Jul 09 18:32:20.530988 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.530964    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:32:20.670325 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.669706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tmp-dir\") pod \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") "
Jul 09 18:32:20.670325 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.669774    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl27w\" (UniqueName: \"kubernetes.io/projected/8abdb8cd-36f1-42de-a65e-ed6af385fac7-kube-api-access-hl27w\") pod \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") "
Jul 09 18:32:20.670325 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.669826    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tls-certs\") pod \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") "
Jul 09 18:32:20.670325 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.669883    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-dshm\") pod \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") "
Jul 09 18:32:20.673067 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.672862    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-dshm" (OuterVolumeSpecName: "dshm") pod "8abdb8cd-36f1-42de-a65e-ed6af385fac7" (UID: "8abdb8cd-36f1-42de-a65e-ed6af385fac7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:20.673525 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.673244    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8abdb8cd-36f1-42de-a65e-ed6af385fac7" (UID: "8abdb8cd-36f1-42de-a65e-ed6af385fac7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:20.675649 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.675600    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8abdb8cd-36f1-42de-a65e-ed6af385fac7-kube-api-access-hl27w" (OuterVolumeSpecName: "kube-api-access-hl27w") pod "8abdb8cd-36f1-42de-a65e-ed6af385fac7" (UID: "8abdb8cd-36f1-42de-a65e-ed6af385fac7"). InnerVolumeSpecName "kube-api-access-hl27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:32:20.676443 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.675801    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-home\") pod \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") "
Jul 09 18:32:20.676443 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.675863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-model-cache\") pod \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\" (UID: \"8abdb8cd-36f1-42de-a65e-ed6af385fac7\") "
Jul 09 18:32:20.676443 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.676211    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:32:20.676443 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.676231    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl27w\" (UniqueName: \"kubernetes.io/projected/8abdb8cd-36f1-42de-a65e-ed6af385fac7-kube-api-access-hl27w\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:32:20.676443 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.676246    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:32:20.676957 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.676764    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8abdb8cd-36f1-42de-a65e-ed6af385fac7" (UID: "8abdb8cd-36f1-42de-a65e-ed6af385fac7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:20.679884 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.677078    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-home" (OuterVolumeSpecName: "home") pod "8abdb8cd-36f1-42de-a65e-ed6af385fac7" (UID: "8abdb8cd-36f1-42de-a65e-ed6af385fac7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:32:20.686914 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.686882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8abdb8cd-36f1-42de-a65e-ed6af385fac7" (UID: "8abdb8cd-36f1-42de-a65e-ed6af385fac7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:32:20.777603 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.777560    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8abdb8cd-36f1-42de-a65e-ed6af385fac7-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:32:20.777603 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.777602    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:32:20.777792 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:20.777618    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8abdb8cd-36f1-42de-a65e-ed6af385fac7-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:32:21.139847 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.139801    2577 generic.go:356] "Generic (PLEG): container finished" podID="8abdb8cd-36f1-42de-a65e-ed6af385fac7" containerID="bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe" exitCode=0
Jul 09 18:32:21.139986 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.139874    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw" event={"ID":"8abdb8cd-36f1-42de-a65e-ed6af385fac7","Type":"ContainerDied","Data":"bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe"}
Jul 09 18:32:21.139986 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.139954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw" event={"ID":"8abdb8cd-36f1-42de-a65e-ed6af385fac7","Type":"ContainerDied","Data":"1931800e70f0794d435cb37e2a894cc4949de0786d40050a7c1a831ef18e574a"}
Jul 09 18:32:21.139986 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.139884    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"
Jul 09 18:32:21.140108 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.139985    2577 scope.go:117] "RemoveContainer" containerID="bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe"
Jul 09 18:32:21.149770 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.149749    2577 scope.go:117] "RemoveContainer" containerID="bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe"
Jul 09 18:32:21.150049 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:32:21.150028    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe\": container with ID starting with bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe not found: ID does not exist" containerID="bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe"
Jul 09 18:32:21.150111 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.150057    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe"} err="failed to get container status \"bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe\": rpc error: code = NotFound desc = could not find container \"bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe\": container with ID starting with bfeb0e3b6f2932ca8572c26a33dc1288c5416e1c6a21e66955067df12a772cfe not found: ID does not exist"
Jul 09 18:32:21.163847 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.163815    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"]
Jul 09 18:32:21.167925 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:21.167903    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-wfsdw"]
Jul 09 18:32:22.417957 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:22.417926    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8abdb8cd-36f1-42de-a65e-ed6af385fac7" path="/var/lib/kubelet/pods/8abdb8cd-36f1-42de-a65e-ed6af385fac7/volumes"
Jul 09 18:32:27.736885 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:27.736821    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:32:36.703664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.703621    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"]
Jul 09 18:32:36.704310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.704088    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8abdb8cd-36f1-42de-a65e-ed6af385fac7" containerName="main"
Jul 09 18:32:36.704310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.704107    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8abdb8cd-36f1-42de-a65e-ed6af385fac7" containerName="main"
Jul 09 18:32:36.704310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.704187    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8abdb8cd-36f1-42de-a65e-ed6af385fac7" containerName="main"
Jul 09 18:32:36.708022 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.707996    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.711405 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.711385    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 09 18:32:36.715732 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.715704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"]
Jul 09 18:32:36.801945 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.801910    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.802104 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.802007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.802104 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.802048    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lrr\" (UniqueName: \"kubernetes.io/projected/bff6c8cd-3979-421f-b447-dfa1f36f8a04-kube-api-access-k9lrr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.802204 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.802123    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.802204 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.802152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.802204 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.802185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.902768 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.902732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.902970 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.902879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.902970 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.902916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lrr\" (UniqueName: \"kubernetes.io/projected/bff6c8cd-3979-421f-b447-dfa1f36f8a04-kube-api-access-k9lrr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.902970 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.902962    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.903170 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.902989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.903170 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.903021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.903622 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.903591    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.903796 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.903654    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.903796 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.903704    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.905689 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.905661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.905925 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.905905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:36.923117 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:36.923089    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9lrr\" (UniqueName: \"kubernetes.io/projected/bff6c8cd-3979-421f-b447-dfa1f36f8a04-kube-api-access-k9lrr\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:37.025770 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:37.025740    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:37.168338 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:37.168302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"]
Jul 09 18:32:37.170549 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:32:37.170526    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff6c8cd_3979_421f_b447_dfa1f36f8a04.slice/crio-06f2eee18af6a2ca2fa9d79152e0a444be6b40105421fbf9f060cfe5cfbabeb2 WatchSource:0}: Error finding container 06f2eee18af6a2ca2fa9d79152e0a444be6b40105421fbf9f060cfe5cfbabeb2: Status 404 returned error can't find the container with id 06f2eee18af6a2ca2fa9d79152e0a444be6b40105421fbf9f060cfe5cfbabeb2
Jul 09 18:32:37.221890 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:37.221856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w" event={"ID":"bff6c8cd-3979-421f-b447-dfa1f36f8a04","Type":"ContainerStarted","Data":"06f2eee18af6a2ca2fa9d79152e0a444be6b40105421fbf9f060cfe5cfbabeb2"}
Jul 09 18:32:37.736588 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:37.736540    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:32:38.228807 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:38.228768    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w" event={"ID":"bff6c8cd-3979-421f-b447-dfa1f36f8a04","Type":"ContainerStarted","Data":"322f440f3f17105318c02a296adaf7dd8eddeb411f462afea9cd1b5f2a7b93a8"}
Jul 09 18:32:38.273661 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:38.273600    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w" podStartSLOduration=2.273583737 podStartE2EDuration="2.273583737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:32:38.267095118 +0000 UTC m=+3192.457455034" watchObservedRunningTime="2026-07-09 18:32:38.273583737 +0000 UTC m=+3192.463943652"
Jul 09 18:32:39.025827 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:39.025784    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:47.025926 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:47.025881    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:47.038198 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:47.038169    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:47.288634 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:47.288547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:32:47.737228 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:47.737137    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:32:57.737066 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:32:57.737020    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 18:33:07.746651 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:07.746621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:33:07.754498 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:07.754478    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:33:10.306113 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:10.306076    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"]
Jul 09 18:33:12.923420 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:12.923380    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"]
Jul 09 18:33:25.327444 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.327400    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w" podUID="bff6c8cd-3979-421f-b447-dfa1f36f8a04" containerName="main" containerID="cri-o://322f440f3f17105318c02a296adaf7dd8eddeb411f462afea9cd1b5f2a7b93a8" gracePeriod=45
Jul 09 18:33:25.462071 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.462043    2577 generic.go:356] "Generic (PLEG): container finished" podID="bff6c8cd-3979-421f-b447-dfa1f36f8a04" containerID="322f440f3f17105318c02a296adaf7dd8eddeb411f462afea9cd1b5f2a7b93a8" exitCode=0
Jul 09 18:33:25.462211 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.462119    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w" event={"ID":"bff6c8cd-3979-421f-b447-dfa1f36f8a04","Type":"ContainerDied","Data":"322f440f3f17105318c02a296adaf7dd8eddeb411f462afea9cd1b5f2a7b93a8"}
Jul 09 18:33:25.569210 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.569187    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:33:25.696119 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696037    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tmp-dir\") pod \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") "
Jul 09 18:33:25.696119 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696084    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-home\") pod \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") "
Jul 09 18:33:25.696119 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696120    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-model-cache\") pod \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") "
Jul 09 18:33:25.696397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696148    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-dshm\") pod \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") "
Jul 09 18:33:25.696397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696168    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9lrr\" (UniqueName: \"kubernetes.io/projected/bff6c8cd-3979-421f-b447-dfa1f36f8a04-kube-api-access-k9lrr\") pod \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") "
Jul 09 18:33:25.696397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tls-certs\") pod \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\" (UID: \"bff6c8cd-3979-421f-b447-dfa1f36f8a04\") "
Jul 09 18:33:25.696397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696369    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bff6c8cd-3979-421f-b447-dfa1f36f8a04" (UID: "bff6c8cd-3979-421f-b447-dfa1f36f8a04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:25.696397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696395    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-home" (OuterVolumeSpecName: "home") pod "bff6c8cd-3979-421f-b447-dfa1f36f8a04" (UID: "bff6c8cd-3979-421f-b447-dfa1f36f8a04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:25.696653 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-model-cache" (OuterVolumeSpecName: "model-cache") pod "bff6c8cd-3979-421f-b447-dfa1f36f8a04" (UID: "bff6c8cd-3979-421f-b447-dfa1f36f8a04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:25.696653 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696455    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:25.696653 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.696473    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:25.698328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.698302    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bff6c8cd-3979-421f-b447-dfa1f36f8a04" (UID: "bff6c8cd-3979-421f-b447-dfa1f36f8a04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:33:25.698468 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.698326    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-dshm" (OuterVolumeSpecName: "dshm") pod "bff6c8cd-3979-421f-b447-dfa1f36f8a04" (UID: "bff6c8cd-3979-421f-b447-dfa1f36f8a04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:25.698468 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.698441    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff6c8cd-3979-421f-b447-dfa1f36f8a04-kube-api-access-k9lrr" (OuterVolumeSpecName: "kube-api-access-k9lrr") pod "bff6c8cd-3979-421f-b447-dfa1f36f8a04" (UID: "bff6c8cd-3979-421f-b447-dfa1f36f8a04"). InnerVolumeSpecName "kube-api-access-k9lrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:33:25.796917 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.796885    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:25.796917 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.796917    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bff6c8cd-3979-421f-b447-dfa1f36f8a04-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:25.797090 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.796934    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lrr\" (UniqueName: \"kubernetes.io/projected/bff6c8cd-3979-421f-b447-dfa1f36f8a04-kube-api-access-k9lrr\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:25.797090 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:25.796948    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bff6c8cd-3979-421f-b447-dfa1f36f8a04-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:26.467702 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:26.467671    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w" event={"ID":"bff6c8cd-3979-421f-b447-dfa1f36f8a04","Type":"ContainerDied","Data":"06f2eee18af6a2ca2fa9d79152e0a444be6b40105421fbf9f060cfe5cfbabeb2"}
Jul 09 18:33:26.467702 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:26.467693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"
Jul 09 18:33:26.468193 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:26.467728    2577 scope.go:117] "RemoveContainer" containerID="322f440f3f17105318c02a296adaf7dd8eddeb411f462afea9cd1b5f2a7b93a8"
Jul 09 18:33:26.488345 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:26.488319    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"]
Jul 09 18:33:26.493649 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:26.493627    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-797dfc7ff4h5s4w"]
Jul 09 18:33:27.947570 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:27.947510    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main" containerID="cri-o://9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c" gracePeriod=45
Jul 09 18:33:28.418408 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.418372    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bff6c8cd-3979-421f-b447-dfa1f36f8a04" path="/var/lib/kubelet/pods/bff6c8cd-3979-421f-b447-dfa1f36f8a04/volumes"
Jul 09 18:33:28.684924 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.684892    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:33:28.823462 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823431    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-model-cache\") pod \"8c501599-ef57-475f-8228-bc61218f7e8a\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") "
Jul 09 18:33:28.823635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-home\") pod \"8c501599-ef57-475f-8228-bc61218f7e8a\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") "
Jul 09 18:33:28.823635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823520    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c501599-ef57-475f-8228-bc61218f7e8a-tls-certs\") pod \"8c501599-ef57-475f-8228-bc61218f7e8a\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") "
Jul 09 18:33:28.823635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823548    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-kserve-provision-location\") pod \"8c501599-ef57-475f-8228-bc61218f7e8a\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") "
Jul 09 18:33:28.823635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823565    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j27cm\" (UniqueName: \"kubernetes.io/projected/8c501599-ef57-475f-8228-bc61218f7e8a-kube-api-access-j27cm\") pod \"8c501599-ef57-475f-8228-bc61218f7e8a\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") "
Jul 09 18:33:28.823635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823587    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-tmp-dir\") pod \"8c501599-ef57-475f-8228-bc61218f7e8a\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") "
Jul 09 18:33:28.823962 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823686    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "8c501599-ef57-475f-8228-bc61218f7e8a" (UID: "8c501599-ef57-475f-8228-bc61218f7e8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:28.823962 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.823766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-dshm\") pod \"8c501599-ef57-475f-8228-bc61218f7e8a\" (UID: \"8c501599-ef57-475f-8228-bc61218f7e8a\") "
Jul 09 18:33:28.824079 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.824034    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:28.824231 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.824196    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-home" (OuterVolumeSpecName: "home") pod "8c501599-ef57-475f-8228-bc61218f7e8a" (UID: "8c501599-ef57-475f-8228-bc61218f7e8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:28.825754 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.825731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-dshm" (OuterVolumeSpecName: "dshm") pod "8c501599-ef57-475f-8228-bc61218f7e8a" (UID: "8c501599-ef57-475f-8228-bc61218f7e8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:28.826201 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.826175    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c501599-ef57-475f-8228-bc61218f7e8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c501599-ef57-475f-8228-bc61218f7e8a" (UID: "8c501599-ef57-475f-8228-bc61218f7e8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:33:28.826294 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.826201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c501599-ef57-475f-8228-bc61218f7e8a-kube-api-access-j27cm" (OuterVolumeSpecName: "kube-api-access-j27cm") pod "8c501599-ef57-475f-8228-bc61218f7e8a" (UID: "8c501599-ef57-475f-8228-bc61218f7e8a"). InnerVolumeSpecName "kube-api-access-j27cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:33:28.843421 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.843377    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8c501599-ef57-475f-8228-bc61218f7e8a" (UID: "8c501599-ef57-475f-8228-bc61218f7e8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:28.890563 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.890540    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c501599-ef57-475f-8228-bc61218f7e8a" (UID: "8c501599-ef57-475f-8228-bc61218f7e8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:33:28.924658 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.924610    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:28.924658 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.924632    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c501599-ef57-475f-8228-bc61218f7e8a-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:28.924658 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.924643    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:28.924658 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.924652    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j27cm\" (UniqueName: \"kubernetes.io/projected/8c501599-ef57-475f-8228-bc61218f7e8a-kube-api-access-j27cm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:28.924828 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.924661    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:28.924828 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:28.924670    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c501599-ef57-475f-8228-bc61218f7e8a-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:33:29.485233 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.485197    2577 generic.go:356] "Generic (PLEG): container finished" podID="8c501599-ef57-475f-8228-bc61218f7e8a" containerID="9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c" exitCode=0
Jul 09 18:33:29.485664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.485239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" event={"ID":"8c501599-ef57-475f-8228-bc61218f7e8a","Type":"ContainerDied","Data":"9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c"}
Jul 09 18:33:29.485664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.485248    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"
Jul 09 18:33:29.485664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.485285    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2" event={"ID":"8c501599-ef57-475f-8228-bc61218f7e8a","Type":"ContainerDied","Data":"09dab482c1904f6bc771312bc97a40675fd90cda0eadfbee0102c8a253b92df5"}
Jul 09 18:33:29.485664 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.485310    2577 scope.go:117] "RemoveContainer" containerID="9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c"
Jul 09 18:33:29.494290 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.494274    2577 scope.go:117] "RemoveContainer" containerID="218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9"
Jul 09 18:33:29.501573 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.501556    2577 scope.go:117] "RemoveContainer" containerID="9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c"
Jul 09 18:33:29.501804 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:33:29.501786    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c\": container with ID starting with 9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c not found: ID does not exist" containerID="9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c"
Jul 09 18:33:29.501872 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.501813    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c"} err="failed to get container status \"9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c\": rpc error: code = NotFound desc = could not find container \"9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c\": container with ID starting with 9d8b1f4e386e3d5667570d218a1b04c61435bfd7e7fc400a9e165b900f6c240c not found: ID does not exist"
Jul 09 18:33:29.501939 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.501830    2577 scope.go:117] "RemoveContainer" containerID="218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9"
Jul 09 18:33:29.502103 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:33:29.502082    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9\": container with ID starting with 218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9 not found: ID does not exist" containerID="218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9"
Jul 09 18:33:29.502143 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.502107    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9"} err="failed to get container status \"218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9\": rpc error: code = NotFound desc = could not find container \"218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9\": container with ID starting with 218529fe23d125e9dae68a08076e79501b75fe62e9110b772841e5121b3924e9 not found: ID does not exist"
Jul 09 18:33:29.515701 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.515681    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"]
Jul 09 18:33:29.518080 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:29.518060    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-6799cd4648-xlzj2"]
Jul 09 18:33:30.418193 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:30.418162    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" path="/var/lib/kubelet/pods/8c501599-ef57-475f-8228-bc61218f7e8a/volumes"
Jul 09 18:33:36.197638 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.197600    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"]
Jul 09 18:33:36.198150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198087    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bff6c8cd-3979-421f-b447-dfa1f36f8a04" containerName="main"
Jul 09 18:33:36.198150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198105    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bff6c8cd-3979-421f-b447-dfa1f36f8a04" containerName="main"
Jul 09 18:33:36.198150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198136    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="storage-initializer"
Jul 09 18:33:36.198150 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198146    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="storage-initializer"
Jul 09 18:33:36.198396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198165    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main"
Jul 09 18:33:36.198396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198175    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main"
Jul 09 18:33:36.198396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198249    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c501599-ef57-475f-8228-bc61218f7e8a" containerName="main"
Jul 09 18:33:36.198396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.198261    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bff6c8cd-3979-421f-b447-dfa1f36f8a04" containerName="main"
Jul 09 18:33:36.203401 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.203380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.206094 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.206071    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 09 18:33:36.210474 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.210453    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"]
Jul 09 18:33:36.378106 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.378077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b8fb60-9168-4b31-af91-e4ae467546c5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.378310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.378125    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.378310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.378166    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvxp7\" (UniqueName: \"kubernetes.io/projected/e5b8fb60-9168-4b31-af91-e4ae467546c5-kube-api-access-zvxp7\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.378310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.378247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.378458 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.378361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.378458 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.378381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479136 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b8fb60-9168-4b31-af91-e4ae467546c5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479136 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479352 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvxp7\" (UniqueName: \"kubernetes.io/projected/e5b8fb60-9168-4b31-af91-e4ae467546c5-kube-api-access-zvxp7\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479352 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479169    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479352 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479352 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479622 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479596    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479715 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.479780 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.479751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.481758 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.481732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.482023 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.482002    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b8fb60-9168-4b31-af91-e4ae467546c5-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.490658 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.490631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvxp7\" (UniqueName: \"kubernetes.io/projected/e5b8fb60-9168-4b31-af91-e4ae467546c5-kube-api-access-zvxp7\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.513643 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.513617    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:36.654948 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:36.654926    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"]
Jul 09 18:33:36.656888 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:33:36.656865    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b8fb60_9168_4b31_af91_e4ae467546c5.slice/crio-741088b7c533edeae5f5b35df049b74aa2747a0e699f14dd51f1e14d17b98287 WatchSource:0}: Error finding container 741088b7c533edeae5f5b35df049b74aa2747a0e699f14dd51f1e14d17b98287: Status 404 returned error can't find the container with id 741088b7c533edeae5f5b35df049b74aa2747a0e699f14dd51f1e14d17b98287
Jul 09 18:33:37.525343 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:37.525309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd" event={"ID":"e5b8fb60-9168-4b31-af91-e4ae467546c5","Type":"ContainerStarted","Data":"8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b"}
Jul 09 18:33:37.525792 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:37.525356    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd" event={"ID":"e5b8fb60-9168-4b31-af91-e4ae467546c5","Type":"ContainerStarted","Data":"741088b7c533edeae5f5b35df049b74aa2747a0e699f14dd51f1e14d17b98287"}
Jul 09 18:33:37.546945 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:37.546900    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd" podStartSLOduration=1.5468848899999998 podStartE2EDuration="1.54688489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:37.545817936 +0000 UTC m=+3251.736177852" watchObservedRunningTime="2026-07-09 18:33:37.54688489 +0000 UTC m=+3251.737244830"
Jul 09 18:33:38.514079 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:38.514039    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:46.513920 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:46.513887    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:46.526620 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:46.526595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:46.577961 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:46.577934    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:33:48.893851 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:48.893800    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"]
Jul 09 18:33:48.897277 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:48.897258    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:48.900709 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:48.900687    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 09 18:33:48.913749 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:48.913724    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"]
Jul 09 18:33:49.077120 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.077097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tmp-dir\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.077272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.077132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldzxk\" (UniqueName: \"kubernetes.io/projected/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kube-api-access-ldzxk\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.077272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.077170    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-dshm\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.077272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.077223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-home\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.077272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.077268    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.077429 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.077340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-model-cache\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.077429 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.077383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tls-certs\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tmp-dir\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldzxk\" (UniqueName: \"kubernetes.io/projected/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kube-api-access-ldzxk\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-dshm\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-home\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178674 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178674 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178391    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-model-cache\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178674 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tls-certs\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178674 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tmp-dir\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.178922 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.178885    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-model-cache\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.179058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.179033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.179154 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.179121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-home\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.180614 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.180590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-dshm\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.181270 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.181238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tls-certs\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.192852 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.192814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldzxk\" (UniqueName: \"kubernetes.io/projected/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kube-api-access-ldzxk\") pod \"prestop-hook-test-kserve-5c66d6b7c6-wq7jj\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.207535 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.207510    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:49.358885 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.358859    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"]
Jul 09 18:33:49.361873 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:33:49.361822    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292d0f2e_f7f2_4870_b247_a3b21ff4b27c.slice/crio-dc130f464b4f50696385d2f17fc0b5479bd429f0156bb6241a47ee8fde77f3fb WatchSource:0}: Error finding container dc130f464b4f50696385d2f17fc0b5479bd429f0156bb6241a47ee8fde77f3fb: Status 404 returned error can't find the container with id dc130f464b4f50696385d2f17fc0b5479bd429f0156bb6241a47ee8fde77f3fb
Jul 09 18:33:49.581120 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.581090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" event={"ID":"292d0f2e-f7f2-4870-b247-a3b21ff4b27c","Type":"ContainerStarted","Data":"69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a"}
Jul 09 18:33:49.581255 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:49.581140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" event={"ID":"292d0f2e-f7f2-4870-b247-a3b21ff4b27c","Type":"ContainerStarted","Data":"dc130f464b4f50696385d2f17fc0b5479bd429f0156bb6241a47ee8fde77f3fb"}
Jul 09 18:33:53.603986 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:53.603952    2577 generic.go:356] "Generic (PLEG): container finished" podID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerID="69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a" exitCode=0
Jul 09 18:33:53.604454 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:53.603999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" event={"ID":"292d0f2e-f7f2-4870-b247-a3b21ff4b27c","Type":"ContainerDied","Data":"69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a"}
Jul 09 18:33:54.610319 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:54.610279    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" event={"ID":"292d0f2e-f7f2-4870-b247-a3b21ff4b27c","Type":"ContainerStarted","Data":"1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c"}
Jul 09 18:33:54.638617 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:54.638558    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podStartSLOduration=3.2112571340000002 podStartE2EDuration="6.638538069s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.427280935s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:33:54.634507414 +0000 UTC m=+3268.824867330" watchObservedRunningTime="2026-07-09 18:33:54.638538069 +0000 UTC m=+3268.828897984"
Jul 09 18:33:55.208306 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:55.208259    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:59.207716 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:59.207682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:33:59.209393 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:33:59.209367    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:34:09.208266 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:09.208223    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:34:10.735462 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:10.735429    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"]
Jul 09 18:34:19.209138 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:19.209073    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:34:25.758873 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:25.758815    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd" podUID="e5b8fb60-9168-4b31-af91-e4ae467546c5" containerName="main" containerID="cri-o://8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b" gracePeriod=45
Jul 09 18:34:26.009036 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.008981    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:34:26.096087 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096052    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-tmp-dir\") pod \"e5b8fb60-9168-4b31-af91-e4ae467546c5\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") "
Jul 09 18:34:26.096269 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b8fb60-9168-4b31-af91-e4ae467546c5-tls-certs\") pod \"e5b8fb60-9168-4b31-af91-e4ae467546c5\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") "
Jul 09 18:34:26.096269 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096137    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvxp7\" (UniqueName: \"kubernetes.io/projected/e5b8fb60-9168-4b31-af91-e4ae467546c5-kube-api-access-zvxp7\") pod \"e5b8fb60-9168-4b31-af91-e4ae467546c5\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") "
Jul 09 18:34:26.096269 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-home\") pod \"e5b8fb60-9168-4b31-af91-e4ae467546c5\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") "
Jul 09 18:34:26.096269 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-model-cache\") pod \"e5b8fb60-9168-4b31-af91-e4ae467546c5\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") "
Jul 09 18:34:26.096269 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-dshm\") pod \"e5b8fb60-9168-4b31-af91-e4ae467546c5\" (UID: \"e5b8fb60-9168-4b31-af91-e4ae467546c5\") "
Jul 09 18:34:26.096617 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096373    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5b8fb60-9168-4b31-af91-e4ae467546c5" (UID: "e5b8fb60-9168-4b31-af91-e4ae467546c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:26.096617 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096547    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:34:26.096729 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096638    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5b8fb60-9168-4b31-af91-e4ae467546c5" (UID: "e5b8fb60-9168-4b31-af91-e4ae467546c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:26.096729 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.096711    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-home" (OuterVolumeSpecName: "home") pod "e5b8fb60-9168-4b31-af91-e4ae467546c5" (UID: "e5b8fb60-9168-4b31-af91-e4ae467546c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:26.098317 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.098280    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-dshm" (OuterVolumeSpecName: "dshm") pod "e5b8fb60-9168-4b31-af91-e4ae467546c5" (UID: "e5b8fb60-9168-4b31-af91-e4ae467546c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:34:26.098729 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.098692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b8fb60-9168-4b31-af91-e4ae467546c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5b8fb60-9168-4b31-af91-e4ae467546c5" (UID: "e5b8fb60-9168-4b31-af91-e4ae467546c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:34:26.098880 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.098721    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b8fb60-9168-4b31-af91-e4ae467546c5-kube-api-access-zvxp7" (OuterVolumeSpecName: "kube-api-access-zvxp7") pod "e5b8fb60-9168-4b31-af91-e4ae467546c5" (UID: "e5b8fb60-9168-4b31-af91-e4ae467546c5"). InnerVolumeSpecName "kube-api-access-zvxp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:34:26.197107 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.197070    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:34:26.197107 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.197097    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:34:26.197107 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.197106    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5b8fb60-9168-4b31-af91-e4ae467546c5-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:34:26.197107 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.197116    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5b8fb60-9168-4b31-af91-e4ae467546c5-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:34:26.197480 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.197125    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvxp7\" (UniqueName: \"kubernetes.io/projected/e5b8fb60-9168-4b31-af91-e4ae467546c5-kube-api-access-zvxp7\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:34:26.467272 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.467166    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:34:26.475830 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.475812    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:34:26.768483 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.768453    2577 generic.go:356] "Generic (PLEG): container finished" podID="e5b8fb60-9168-4b31-af91-e4ae467546c5" containerID="8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b" exitCode=0
Jul 09 18:34:26.768918 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.768510    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"
Jul 09 18:34:26.768918 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.768531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd" event={"ID":"e5b8fb60-9168-4b31-af91-e4ae467546c5","Type":"ContainerDied","Data":"8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b"}
Jul 09 18:34:26.768918 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.768583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd" event={"ID":"e5b8fb60-9168-4b31-af91-e4ae467546c5","Type":"ContainerDied","Data":"741088b7c533edeae5f5b35df049b74aa2747a0e699f14dd51f1e14d17b98287"}
Jul 09 18:34:26.768918 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.768607    2577 scope.go:117] "RemoveContainer" containerID="8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b"
Jul 09 18:34:26.778056 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.778039    2577 scope.go:117] "RemoveContainer" containerID="8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b"
Jul 09 18:34:26.778317 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:34:26.778298    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b\": container with ID starting with 8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b not found: ID does not exist" containerID="8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b"
Jul 09 18:34:26.778371 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.778329    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b"} err="failed to get container status \"8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b\": rpc error: code = NotFound desc = could not find container \"8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b\": container with ID starting with 8c26ae554a178d81c1bd4e8037454ff893b3e9b81a971af444eab1e570aef61b not found: ID does not exist"
Jul 09 18:34:26.787273 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.787247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"]
Jul 09 18:34:26.790556 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:26.790532    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-p9jvd"]
Jul 09 18:34:28.420283 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:28.420243    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5b8fb60-9168-4b31-af91-e4ae467546c5" path="/var/lib/kubelet/pods/e5b8fb60-9168-4b31-af91-e4ae467546c5/volumes"
Jul 09 18:34:29.208242 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:29.208209    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:34:39.208734 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:39.208689    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:34:49.208599 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:49.208512    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:34:54.196030 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.195992    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"]
Jul 09 18:34:54.196513 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.196448    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b8fb60-9168-4b31-af91-e4ae467546c5" containerName="main"
Jul 09 18:34:54.196513 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.196466    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b8fb60-9168-4b31-af91-e4ae467546c5" containerName="main"
Jul 09 18:34:54.196656 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.196550    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b8fb60-9168-4b31-af91-e4ae467546c5" containerName="main"
Jul 09 18:34:54.198899 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.198874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.201584 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.201561    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 09 18:34:54.210290 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.209613    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"]
Jul 09 18:34:54.221812 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.221788    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.221935 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.221845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.221935 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.221872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29be87b-79e0-4043-9413-f03799f1ec1e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.222032 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.221979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.222083 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.222072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j77km\" (UniqueName: \"kubernetes.io/projected/b29be87b-79e0-4043-9413-f03799f1ec1e-kube-api-access-j77km\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.222144 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.222127    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323181 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323147    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323207    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j77km\" (UniqueName: \"kubernetes.io/projected/b29be87b-79e0-4043-9413-f03799f1ec1e-kube-api-access-j77km\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323321    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323356 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29be87b-79e0-4043-9413-f03799f1ec1e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323685 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323832 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.323947 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.323863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.325395 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.325371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.325630 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.325615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29be87b-79e0-4043-9413-f03799f1ec1e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.332117 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.332099    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j77km\" (UniqueName: \"kubernetes.io/projected/b29be87b-79e0-4043-9413-f03799f1ec1e-kube-api-access-j77km\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.514703 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.514671    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:54.685779 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.685749    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"]
Jul 09 18:34:54.688120 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:34:54.688091    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb29be87b_79e0_4043_9413_f03799f1ec1e.slice/crio-299920edfa3921d1d1e566e3f64a23f743591bc65840a6f6d28cc2ee48cb8c9e WatchSource:0}: Error finding container 299920edfa3921d1d1e566e3f64a23f743591bc65840a6f6d28cc2ee48cb8c9e: Status 404 returned error can't find the container with id 299920edfa3921d1d1e566e3f64a23f743591bc65840a6f6d28cc2ee48cb8c9e
Jul 09 18:34:54.906610 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.906521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8" event={"ID":"b29be87b-79e0-4043-9413-f03799f1ec1e","Type":"ContainerStarted","Data":"a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719"}
Jul 09 18:34:54.906610 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.906572    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8" event={"ID":"b29be87b-79e0-4043-9413-f03799f1ec1e","Type":"ContainerStarted","Data":"299920edfa3921d1d1e566e3f64a23f743591bc65840a6f6d28cc2ee48cb8c9e"}
Jul 09 18:34:54.927286 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:54.927242    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8" podStartSLOduration=0.927227574 podStartE2EDuration="927.227574ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:34:54.926043742 +0000 UTC m=+3329.116403695" watchObservedRunningTime="2026-07-09 18:34:54.927227574 +0000 UTC m=+3329.117587489"
Jul 09 18:34:55.515800 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:55.515768    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:34:59.209038 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:34:59.208993    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:35:04.515317 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:04.515279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:35:04.528472 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:04.528443    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:35:04.963397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:04.963314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:35:09.208961 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:09.208919    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:35:19.208647 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:19.208591    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:35:27.578108 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:27.578077    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"]
Jul 09 18:35:29.208560 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:29.208513    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:35:39.208967 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:39.208914    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:35:42.599286 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:42.599217    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8" podUID="b29be87b-79e0-4043-9413-f03799f1ec1e" containerName="main" containerID="cri-o://a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719" gracePeriod=45
Jul 09 18:35:42.858337 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:42.858282    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:35:43.036748 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.036705    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j77km\" (UniqueName: \"kubernetes.io/projected/b29be87b-79e0-4043-9413-f03799f1ec1e-kube-api-access-j77km\") pod \"b29be87b-79e0-4043-9413-f03799f1ec1e\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") "
Jul 09 18:35:43.036935 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.036763    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-home\") pod \"b29be87b-79e0-4043-9413-f03799f1ec1e\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") "
Jul 09 18:35:43.036935 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.036888    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29be87b-79e0-4043-9413-f03799f1ec1e-tls-certs\") pod \"b29be87b-79e0-4043-9413-f03799f1ec1e\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") "
Jul 09 18:35:43.036935 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.036927    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-dshm\") pod \"b29be87b-79e0-4043-9413-f03799f1ec1e\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") "
Jul 09 18:35:43.037110 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.036990    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-model-cache\") pod \"b29be87b-79e0-4043-9413-f03799f1ec1e\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") "
Jul 09 18:35:43.037110 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.037036    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-home" (OuterVolumeSpecName: "home") pod "b29be87b-79e0-4043-9413-f03799f1ec1e" (UID: "b29be87b-79e0-4043-9413-f03799f1ec1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:35:43.037110 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.037044    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-tmp-dir\") pod \"b29be87b-79e0-4043-9413-f03799f1ec1e\" (UID: \"b29be87b-79e0-4043-9413-f03799f1ec1e\") "
Jul 09 18:35:43.037274 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.037205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b29be87b-79e0-4043-9413-f03799f1ec1e" (UID: "b29be87b-79e0-4043-9413-f03799f1ec1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:35:43.037428 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.037398    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b29be87b-79e0-4043-9413-f03799f1ec1e" (UID: "b29be87b-79e0-4043-9413-f03799f1ec1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:35:43.037635 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.037600    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:35:43.037707 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.037649    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:35:43.037707 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.037663    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:35:43.039100 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.039073    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-dshm" (OuterVolumeSpecName: "dshm") pod "b29be87b-79e0-4043-9413-f03799f1ec1e" (UID: "b29be87b-79e0-4043-9413-f03799f1ec1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:35:43.039359 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.039340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29be87b-79e0-4043-9413-f03799f1ec1e-kube-api-access-j77km" (OuterVolumeSpecName: "kube-api-access-j77km") pod "b29be87b-79e0-4043-9413-f03799f1ec1e" (UID: "b29be87b-79e0-4043-9413-f03799f1ec1e"). InnerVolumeSpecName "kube-api-access-j77km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:35:43.039443 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.039429    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b29be87b-79e0-4043-9413-f03799f1ec1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b29be87b-79e0-4043-9413-f03799f1ec1e" (UID: "b29be87b-79e0-4043-9413-f03799f1ec1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:35:43.138761 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.138660    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j77km\" (UniqueName: \"kubernetes.io/projected/b29be87b-79e0-4043-9413-f03799f1ec1e-kube-api-access-j77km\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:35:43.138761 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.138699    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b29be87b-79e0-4043-9413-f03799f1ec1e-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:35:43.138761 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.138716    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b29be87b-79e0-4043-9413-f03799f1ec1e-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:35:43.149577 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.149543    2577 generic.go:356] "Generic (PLEG): container finished" podID="b29be87b-79e0-4043-9413-f03799f1ec1e" containerID="a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719" exitCode=0
Jul 09 18:35:43.149746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.149582    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"
Jul 09 18:35:43.149746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.149644    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8" event={"ID":"b29be87b-79e0-4043-9413-f03799f1ec1e","Type":"ContainerDied","Data":"a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719"}
Jul 09 18:35:43.149746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.149738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8" event={"ID":"b29be87b-79e0-4043-9413-f03799f1ec1e","Type":"ContainerDied","Data":"299920edfa3921d1d1e566e3f64a23f743591bc65840a6f6d28cc2ee48cb8c9e"}
Jul 09 18:35:43.149949 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.149810    2577 scope.go:117] "RemoveContainer" containerID="a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719"
Jul 09 18:35:43.161471 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.161454    2577 scope.go:117] "RemoveContainer" containerID="a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719"
Jul 09 18:35:43.161732 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:35:43.161711    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719\": container with ID starting with a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719 not found: ID does not exist" containerID="a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719"
Jul 09 18:35:43.161885 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.161736    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719"} err="failed to get container status \"a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719\": rpc error: code = NotFound desc = could not find container \"a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719\": container with ID starting with a827a3dad6ca0d500075afb71db7fb2afd1ba2dd4b6c9f6be9ea25591ab58719 not found: ID does not exist"
Jul 09 18:35:43.172497 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.172466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"]
Jul 09 18:35:43.176336 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:43.176316    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-8vkw8"]
Jul 09 18:35:44.419018 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:44.418975    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b29be87b-79e0-4043-9413-f03799f1ec1e" path="/var/lib/kubelet/pods/b29be87b-79e0-4043-9413-f03799f1ec1e/volumes"
Jul 09 18:35:49.218126 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:49.218094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:35:49.226132 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:49.226111    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:35:53.779665 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.779630    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"]
Jul 09 18:35:53.808746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.808717    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"]
Jul 09 18:35:53.809246 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.809217    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b29be87b-79e0-4043-9413-f03799f1ec1e" containerName="main"
Jul 09 18:35:53.809246 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.809243    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b29be87b-79e0-4043-9413-f03799f1ec1e" containerName="main"
Jul 09 18:35:53.809448 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.809326    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b29be87b-79e0-4043-9413-f03799f1ec1e" containerName="main"
Jul 09 18:35:53.812298 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.812274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.816438 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.816412    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tmp-dir\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.816553 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.816472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-dshm\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.816553 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.816519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-model-cache\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.816636 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.816598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-home\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.816680 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.816644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.816727 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.816679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tls-certs\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.816727 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.816709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzv9\" (UniqueName: \"kubernetes.io/projected/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kube-api-access-jxzv9\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.825984 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.825948    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"]
Jul 09 18:35:53.917915 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.917886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-home\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918076 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.917927    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918076 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.917945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tls-certs\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918076 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.917970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzv9\" (UniqueName: \"kubernetes.io/projected/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kube-api-access-jxzv9\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918076 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.918027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tmp-dir\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918076 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.918069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-dshm\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918335 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.918215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-model-cache\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918395 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.918329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918459 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.918398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-home\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918459 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.918410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tmp-dir\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.918697 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.918676    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-model-cache\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.920274 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.920253    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-dshm\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.920451 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.920433    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tls-certs\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:53.927431 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:53.927411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzv9\" (UniqueName: \"kubernetes.io/projected/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kube-api-access-jxzv9\") pod \"prestop-hook-test-kserve-5c66d6b7c6-hnjkg\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:54.124994 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:54.124943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:35:54.286901 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:54.286869    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"]
Jul 09 18:35:54.288701 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:35:54.288675    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d56d97_f92c_4f57_ab37_9b4fca7b6627.slice/crio-d96ccae3b39a8344ba7319f28547e3cb527c41ff30f7d22ef35a91dde8a1f4a4 WatchSource:0}: Error finding container d96ccae3b39a8344ba7319f28547e3cb527c41ff30f7d22ef35a91dde8a1f4a4: Status 404 returned error can't find the container with id d96ccae3b39a8344ba7319f28547e3cb527c41ff30f7d22ef35a91dde8a1f4a4
Jul 09 18:35:55.204736 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:55.204699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" event={"ID":"e2d56d97-f92c-4f57-ab37-9b4fca7b6627","Type":"ContainerStarted","Data":"d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8"}
Jul 09 18:35:55.205234 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:55.204758    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" event={"ID":"e2d56d97-f92c-4f57-ab37-9b4fca7b6627","Type":"ContainerStarted","Data":"d96ccae3b39a8344ba7319f28547e3cb527c41ff30f7d22ef35a91dde8a1f4a4"}
Jul 09 18:35:58.219858 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:58.219811    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerID="d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8" exitCode=0
Jul 09 18:35:58.220238 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:58.219887    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" event={"ID":"e2d56d97-f92c-4f57-ab37-9b4fca7b6627","Type":"ContainerDied","Data":"d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8"}
Jul 09 18:35:59.227973 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:59.227926    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" event={"ID":"e2d56d97-f92c-4f57-ab37-9b4fca7b6627","Type":"ContainerStarted","Data":"d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486"}
Jul 09 18:35:59.252547 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:35:59.252503    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" podStartSLOduration=2.439678859 podStartE2EDuration="6.252487819s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81280896s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:35:59.250876081 +0000 UTC m=+3393.441235999" watchObservedRunningTime="2026-07-09 18:35:59.252487819 +0000 UTC m=+3393.442847734"
Jul 09 18:36:00.125919 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:00.125880    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:36:04.125292 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:04.125260    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:36:04.126681 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:04.126655    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 09 18:36:08.804909 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:08.804850    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" containerID="cri-o://1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c" gracePeriod=45
Jul 09 18:36:09.219504 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.219383    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 09 18:36:09.565011 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.564982    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:36:09.653378 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653351    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-model-cache\") pod \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") "
Jul 09 18:36:09.653533 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653409    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tls-certs\") pod \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") "
Jul 09 18:36:09.653533 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653436    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kserve-provision-location\") pod \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") "
Jul 09 18:36:09.653533 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-dshm\") pod \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") "
Jul 09 18:36:09.653533 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653478    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-home\") pod \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") "
Jul 09 18:36:09.653533 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653508    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tmp-dir\") pod \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") "
Jul 09 18:36:09.653815 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldzxk\" (UniqueName: \"kubernetes.io/projected/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kube-api-access-ldzxk\") pod \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\" (UID: \"292d0f2e-f7f2-4870-b247-a3b21ff4b27c\") "
Jul 09 18:36:09.653815 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653674    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-model-cache" (OuterVolumeSpecName: "model-cache") pod "292d0f2e-f7f2-4870-b247-a3b21ff4b27c" (UID: "292d0f2e-f7f2-4870-b247-a3b21ff4b27c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:09.653815 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.653799    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:09.654365 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.654322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-home" (OuterVolumeSpecName: "home") pod "292d0f2e-f7f2-4870-b247-a3b21ff4b27c" (UID: "292d0f2e-f7f2-4870-b247-a3b21ff4b27c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:09.655974 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.655919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "292d0f2e-f7f2-4870-b247-a3b21ff4b27c" (UID: "292d0f2e-f7f2-4870-b247-a3b21ff4b27c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:09.656091 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.656031    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-dshm" (OuterVolumeSpecName: "dshm") pod "292d0f2e-f7f2-4870-b247-a3b21ff4b27c" (UID: "292d0f2e-f7f2-4870-b247-a3b21ff4b27c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:09.656218 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.656197    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kube-api-access-ldzxk" (OuterVolumeSpecName: "kube-api-access-ldzxk") pod "292d0f2e-f7f2-4870-b247-a3b21ff4b27c" (UID: "292d0f2e-f7f2-4870-b247-a3b21ff4b27c"). InnerVolumeSpecName "kube-api-access-ldzxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:09.672053 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.672000    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "292d0f2e-f7f2-4870-b247-a3b21ff4b27c" (UID: "292d0f2e-f7f2-4870-b247-a3b21ff4b27c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:09.716430 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.716396    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "292d0f2e-f7f2-4870-b247-a3b21ff4b27c" (UID: "292d0f2e-f7f2-4870-b247-a3b21ff4b27c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:09.754433 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.754398    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:09.754433 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.754427    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:09.754433 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.754439    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:09.754652 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.754446    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:09.754652 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.754455    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldzxk\" (UniqueName: \"kubernetes.io/projected/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-kube-api-access-ldzxk\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:09.754652 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:09.754463    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/292d0f2e-f7f2-4870-b247-a3b21ff4b27c-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:10.286952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.286917    2577 generic.go:356] "Generic (PLEG): container finished" podID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerID="1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c" exitCode=0
Jul 09 18:36:10.287391 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.286962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" event={"ID":"292d0f2e-f7f2-4870-b247-a3b21ff4b27c","Type":"ContainerDied","Data":"1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c"}
Jul 09 18:36:10.287391 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.286994    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"
Jul 09 18:36:10.287391 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.287005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj" event={"ID":"292d0f2e-f7f2-4870-b247-a3b21ff4b27c","Type":"ContainerDied","Data":"dc130f464b4f50696385d2f17fc0b5479bd429f0156bb6241a47ee8fde77f3fb"}
Jul 09 18:36:10.287391 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.287027    2577 scope.go:117] "RemoveContainer" containerID="1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c"
Jul 09 18:36:10.296199 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.296173    2577 scope.go:117] "RemoveContainer" containerID="69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a"
Jul 09 18:36:10.305359 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.305339    2577 scope.go:117] "RemoveContainer" containerID="1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c"
Jul 09 18:36:10.305654 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:36:10.305635    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c\": container with ID starting with 1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c not found: ID does not exist" containerID="1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c"
Jul 09 18:36:10.305724 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.305664    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c"} err="failed to get container status \"1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c\": rpc error: code = NotFound desc = could not find container \"1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c\": container with ID starting with 1cbdc69b6ccf7b50493b1326314f82d3b8f069d1c8fbe739df85f6abfcca616c not found: ID does not exist"
Jul 09 18:36:10.305724 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.305681    2577 scope.go:117] "RemoveContainer" containerID="69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a"
Jul 09 18:36:10.305974 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:36:10.305952    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a\": container with ID starting with 69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a not found: ID does not exist" containerID="69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a"
Jul 09 18:36:10.306031 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.305983    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a"} err="failed to get container status \"69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a\": rpc error: code = NotFound desc = could not find container \"69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a\": container with ID starting with 69a3d8530ecde344f980b75562cde5b30fecfdbbb54e0ef05662351e157e8b3a not found: ID does not exist"
Jul 09 18:36:10.313803 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.313779    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"]
Jul 09 18:36:10.318818 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.318790    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-wq7jj"]
Jul 09 18:36:10.418710 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:10.418675    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" path="/var/lib/kubelet/pods/292d0f2e-f7f2-4870-b247-a3b21ff4b27c/volumes"
Jul 09 18:36:14.125933 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:14.125882    2577 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 09 18:36:19.801170 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:19.801139    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"]
Jul 09 18:36:34.825555 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:34.825516    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="main" containerID="cri-o://d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486" gracePeriod=45
Jul 09 18:36:36.558070 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.558048    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5c66d6b7c6-hnjkg_e2d56d97-f92c-4f57-ab37-9b4fca7b6627/main/0.log"
Jul 09 18:36:36.559588 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.559570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:36:36.680734 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.680641    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-dshm\") pod \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") "
Jul 09 18:36:36.680734 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.680693    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tmp-dir\") pod \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") "
Jul 09 18:36:36.680734 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.680730    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-model-cache\") pod \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") "
Jul 09 18:36:36.681058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.680753    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-home\") pod \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") "
Jul 09 18:36:36.681058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.680855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kserve-provision-location\") pod \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") "
Jul 09 18:36:36.681058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.680889    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxzv9\" (UniqueName: \"kubernetes.io/projected/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kube-api-access-jxzv9\") pod \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") "
Jul 09 18:36:36.681058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.680930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tls-certs\") pod \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\" (UID: \"e2d56d97-f92c-4f57-ab37-9b4fca7b6627\") "
Jul 09 18:36:36.681058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.681024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-home" (OuterVolumeSpecName: "home") pod "e2d56d97-f92c-4f57-ab37-9b4fca7b6627" (UID: "e2d56d97-f92c-4f57-ab37-9b4fca7b6627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:36.681058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.681053    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2d56d97-f92c-4f57-ab37-9b4fca7b6627" (UID: "e2d56d97-f92c-4f57-ab37-9b4fca7b6627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:36.681384 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.681113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2d56d97-f92c-4f57-ab37-9b4fca7b6627" (UID: "e2d56d97-f92c-4f57-ab37-9b4fca7b6627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:36.681384 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.681154    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:36.681384 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.681172    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:36.683109 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.683084    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2d56d97-f92c-4f57-ab37-9b4fca7b6627" (UID: "e2d56d97-f92c-4f57-ab37-9b4fca7b6627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:36:36.683253 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.683192    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-dshm" (OuterVolumeSpecName: "dshm") pod "e2d56d97-f92c-4f57-ab37-9b4fca7b6627" (UID: "e2d56d97-f92c-4f57-ab37-9b4fca7b6627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:36.683253 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.683204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kube-api-access-jxzv9" (OuterVolumeSpecName: "kube-api-access-jxzv9") pod "e2d56d97-f92c-4f57-ab37-9b4fca7b6627" (UID: "e2d56d97-f92c-4f57-ab37-9b4fca7b6627"). InnerVolumeSpecName "kube-api-access-jxzv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:36:36.744208 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.744180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2d56d97-f92c-4f57-ab37-9b4fca7b6627" (UID: "e2d56d97-f92c-4f57-ab37-9b4fca7b6627"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:36:36.781848 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.781821    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:36.781959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.781889    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:36.781959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.781906    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kserve-provision-location\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:36.781959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.781921    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxzv9\" (UniqueName: \"kubernetes.io/projected/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-kube-api-access-jxzv9\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:36.781959 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:36.781934    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d56d97-f92c-4f57-ab37-9b4fca7b6627-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:36:37.420586 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.420558    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5c66d6b7c6-hnjkg_e2d56d97-f92c-4f57-ab37-9b4fca7b6627/main/0.log"
Jul 09 18:36:37.420961 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.420935    2577 generic.go:356] "Generic (PLEG): container finished" podID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerID="d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486" exitCode=130
Jul 09 18:36:37.421036 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.420982    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"
Jul 09 18:36:37.421036 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.421009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" event={"ID":"e2d56d97-f92c-4f57-ab37-9b4fca7b6627","Type":"ContainerDied","Data":"d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486"}
Jul 09 18:36:37.421106 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.421051    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg" event={"ID":"e2d56d97-f92c-4f57-ab37-9b4fca7b6627","Type":"ContainerDied","Data":"d96ccae3b39a8344ba7319f28547e3cb527c41ff30f7d22ef35a91dde8a1f4a4"}
Jul 09 18:36:37.421106 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.421076    2577 scope.go:117] "RemoveContainer" containerID="d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486"
Jul 09 18:36:37.430174 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.430157    2577 scope.go:117] "RemoveContainer" containerID="d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8"
Jul 09 18:36:37.437985 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.437968    2577 scope.go:117] "RemoveContainer" containerID="d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486"
Jul 09 18:36:37.438231 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:36:37.438211    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486\": container with ID starting with d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486 not found: ID does not exist" containerID="d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486"
Jul 09 18:36:37.438280 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.438240    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486"} err="failed to get container status \"d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486\": rpc error: code = NotFound desc = could not find container \"d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486\": container with ID starting with d81ec7e4818c77e6230d8c5b99a4aacfde3c6a03339bff901efa5d5c653f2486 not found: ID does not exist"
Jul 09 18:36:37.438280 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.438257    2577 scope.go:117] "RemoveContainer" containerID="d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8"
Jul 09 18:36:37.438462 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:36:37.438447    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8\": container with ID starting with d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8 not found: ID does not exist" containerID="d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8"
Jul 09 18:36:37.438502 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.438472    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8"} err="failed to get container status \"d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8\": rpc error: code = NotFound desc = could not find container \"d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8\": container with ID starting with d670ca54773dad02e03d4af6af952f4cc3059730255b38b18a7fc5c201c918a8 not found: ID does not exist"
Jul 09 18:36:37.446727 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.446706    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"]
Jul 09 18:36:37.450806 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:37.450785    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c66d6b7c6-hnjkg"]
Jul 09 18:36:38.417954 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:36:38.417924    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" path="/var/lib/kubelet/pods/e2d56d97-f92c-4f57-ab37-9b4fca7b6627/volumes"
Jul 09 18:37:01.108161 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108127    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"]
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108413    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="storage-initializer"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108424    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="storage-initializer"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108435    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="storage-initializer"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108440    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="storage-initializer"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108448    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="main"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108454    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="main"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108464    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108469    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108513    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2d56d97-f92c-4f57-ab37-9b4fca7b6627" containerName="main"
Jul 09 18:37:01.108548 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.108523    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="292d0f2e-f7f2-4870-b247-a3b21ff4b27c" containerName="main"
Jul 09 18:37:01.111560 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.111544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.114396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.114372    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 09 18:37:01.124820 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.124796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"]
Jul 09 18:37:01.149794 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.149770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c5kb\" (UniqueName: \"kubernetes.io/projected/10c5aa67-b302-40b9-8b6c-27a0129d94bf-kube-api-access-9c5kb\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.149913 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.149831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.149913 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.149903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.150035 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.149926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.150035 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.149960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.150035 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.149995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251119 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251220 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251133    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251220 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251220 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251220 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251425 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251320    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c5kb\" (UniqueName: \"kubernetes.io/projected/10c5aa67-b302-40b9-8b6c-27a0129d94bf-kube-api-access-9c5kb\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251518 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251601 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.251874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.251854    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.253879 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.253861    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.254042 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.254027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.261169 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.261147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c5kb\" (UniqueName: \"kubernetes.io/projected/10c5aa67-b302-40b9-8b6c-27a0129d94bf-kube-api-access-9c5kb\") pod \"rolling-upgrade-test-kserve-54bc76f55b-45st6\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.421425 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.421325    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:01.575221 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:01.575194    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"]
Jul 09 18:37:01.577785 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:37:01.577759    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c5aa67_b302_40b9_8b6c_27a0129d94bf.slice/crio-980e4835de02d5ab721b42b49b30032bc56636038775d65442b4701f91f04fb0 WatchSource:0}: Error finding container 980e4835de02d5ab721b42b49b30032bc56636038775d65442b4701f91f04fb0: Status 404 returned error can't find the container with id 980e4835de02d5ab721b42b49b30032bc56636038775d65442b4701f91f04fb0
Jul 09 18:37:02.538737 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:02.538697    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6" event={"ID":"10c5aa67-b302-40b9-8b6c-27a0129d94bf","Type":"ContainerStarted","Data":"f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586"}
Jul 09 18:37:02.539225 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:02.538751    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6" event={"ID":"10c5aa67-b302-40b9-8b6c-27a0129d94bf","Type":"ContainerStarted","Data":"980e4835de02d5ab721b42b49b30032bc56636038775d65442b4701f91f04fb0"}
Jul 09 18:37:02.562216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:02.562170    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6" podStartSLOduration=1.562155449 podStartE2EDuration="1.562155449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:02.560861118 +0000 UTC m=+3456.751221028" watchObservedRunningTime="2026-07-09 18:37:02.562155449 +0000 UTC m=+3456.752515406"
Jul 09 18:37:03.421943 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:03.421897    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:10.153397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.153363    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"]
Jul 09 18:37:10.156681 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.156656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.159326 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.159303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 09 18:37:10.174002 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.173979    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"]
Jul 09 18:37:10.227325 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.227300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e566e07-6079-435e-a406-7877c04168b8-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.227419 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.227331    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.227419 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.227352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.227419 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.227398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.227550 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.227436    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.227550 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.227460    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdspj\" (UniqueName: \"kubernetes.io/projected/5e566e07-6079-435e-a406-7877c04168b8-kube-api-access-gdspj\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.328737 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.328709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.328874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.328746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdspj\" (UniqueName: \"kubernetes.io/projected/5e566e07-6079-435e-a406-7877c04168b8-kube-api-access-gdspj\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.328874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.328795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e566e07-6079-435e-a406-7877c04168b8-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.328874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.328821    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.329048 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.328944    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.329048 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.328993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.329157 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.329091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.329363 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.329338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.329530 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.329395    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-home\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.331598 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.331567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.331724 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.331706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e566e07-6079-435e-a406-7877c04168b8-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.338373 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.338350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdspj\" (UniqueName: \"kubernetes.io/projected/5e566e07-6079-435e-a406-7877c04168b8-kube-api-access-gdspj\") pod \"scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.466606 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.466508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:10.613309 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:10.613286    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"]
Jul 09 18:37:10.615806 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:37:10.615781    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e566e07_6079_435e_a406_7877c04168b8.slice/crio-cccd2c47588358bb8ffb0c1a64e8c91ff85521561e89016f46725a26acb14395 WatchSource:0}: Error finding container cccd2c47588358bb8ffb0c1a64e8c91ff85521561e89016f46725a26acb14395: Status 404 returned error can't find the container with id cccd2c47588358bb8ffb0c1a64e8c91ff85521561e89016f46725a26acb14395
Jul 09 18:37:11.422556 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:11.422517    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:11.436303 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:11.436278    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:11.582060 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:11.582027    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n" event={"ID":"5e566e07-6079-435e-a406-7877c04168b8","Type":"ContainerStarted","Data":"bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b"}
Jul 09 18:37:11.582231 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:11.582077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n" event={"ID":"5e566e07-6079-435e-a406-7877c04168b8","Type":"ContainerStarted","Data":"cccd2c47588358bb8ffb0c1a64e8c91ff85521561e89016f46725a26acb14395"}
Jul 09 18:37:11.593472 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:11.593449    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:37:11.623750 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:11.623699    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n" podStartSLOduration=1.6236852389999998 podStartE2EDuration="1.623685239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:37:11.620680419 +0000 UTC m=+3465.811040334" watchObservedRunningTime="2026-07-09 18:37:11.623685239 +0000 UTC m=+3465.814045154"
Jul 09 18:37:12.467276 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:12.467240    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:20.467186 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:20.467158    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:20.479632 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:20.479604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:20.631419 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:20.631397    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:37:47.515369 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:47.515330    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"]
Jul 09 18:37:57.201939 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:37:57.201860    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"]
Jul 09 18:38:02.538901 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.538854    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n" podUID="5e566e07-6079-435e-a406-7877c04168b8" containerName="main" containerID="cri-o://bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b" gracePeriod=45
Jul 09 18:38:02.621181 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.621145    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n" podUID="5e566e07-6079-435e-a406-7877c04168b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 09 18:38:02.782770 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.782745    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:38:02.815927 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.815815    2577 generic.go:356] "Generic (PLEG): container finished" podID="5e566e07-6079-435e-a406-7877c04168b8" containerID="bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b" exitCode=0
Jul 09 18:38:02.815927 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.815892    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"
Jul 09 18:38:02.815927 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.815908    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n" event={"ID":"5e566e07-6079-435e-a406-7877c04168b8","Type":"ContainerDied","Data":"bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b"}
Jul 09 18:38:02.816141 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.816002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n" event={"ID":"5e566e07-6079-435e-a406-7877c04168b8","Type":"ContainerDied","Data":"cccd2c47588358bb8ffb0c1a64e8c91ff85521561e89016f46725a26acb14395"}
Jul 09 18:38:02.816141 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.816037    2577 scope.go:117] "RemoveContainer" containerID="bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b"
Jul 09 18:38:02.825171 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.825151    2577 scope.go:117] "RemoveContainer" containerID="bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b"
Jul 09 18:38:02.825427 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:38:02.825408    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b\": container with ID starting with bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b not found: ID does not exist" containerID="bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b"
Jul 09 18:38:02.825499 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.825433    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b"} err="failed to get container status \"bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b\": rpc error: code = NotFound desc = could not find container \"bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b\": container with ID starting with bc5f98b1c9ed903345a59410c79f111249bb9ae15010819a16b90796d15f2d7b not found: ID does not exist"
Jul 09 18:38:02.953449 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953413    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e566e07-6079-435e-a406-7877c04168b8-tls-certs\") pod \"5e566e07-6079-435e-a406-7877c04168b8\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") "
Jul 09 18:38:02.953607 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953517    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-home\") pod \"5e566e07-6079-435e-a406-7877c04168b8\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") "
Jul 09 18:38:02.953607 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953553    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-tmp-dir\") pod \"5e566e07-6079-435e-a406-7877c04168b8\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") "
Jul 09 18:38:02.953607 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953581    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdspj\" (UniqueName: \"kubernetes.io/projected/5e566e07-6079-435e-a406-7877c04168b8-kube-api-access-gdspj\") pod \"5e566e07-6079-435e-a406-7877c04168b8\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") "
Jul 09 18:38:02.953772 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-dshm\") pod \"5e566e07-6079-435e-a406-7877c04168b8\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") "
Jul 09 18:38:02.953772 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953651    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-model-cache\") pod \"5e566e07-6079-435e-a406-7877c04168b8\" (UID: \"5e566e07-6079-435e-a406-7877c04168b8\") "
Jul 09 18:38:02.953904 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953779    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-home" (OuterVolumeSpecName: "home") pod "5e566e07-6079-435e-a406-7877c04168b8" (UID: "5e566e07-6079-435e-a406-7877c04168b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:02.953904 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e566e07-6079-435e-a406-7877c04168b8" (UID: "5e566e07-6079-435e-a406-7877c04168b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:02.954022 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953919    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:02.954022 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953936    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:02.954022 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.953945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e566e07-6079-435e-a406-7877c04168b8" (UID: "5e566e07-6079-435e-a406-7877c04168b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:02.955749 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.955726    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e566e07-6079-435e-a406-7877c04168b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e566e07-6079-435e-a406-7877c04168b8" (UID: "5e566e07-6079-435e-a406-7877c04168b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:38:02.956118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.956097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-dshm" (OuterVolumeSpecName: "dshm") pod "5e566e07-6079-435e-a406-7877c04168b8" (UID: "5e566e07-6079-435e-a406-7877c04168b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:02.956206 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:02.956145    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e566e07-6079-435e-a406-7877c04168b8-kube-api-access-gdspj" (OuterVolumeSpecName: "kube-api-access-gdspj") pod "5e566e07-6079-435e-a406-7877c04168b8" (UID: "5e566e07-6079-435e-a406-7877c04168b8"). InnerVolumeSpecName "kube-api-access-gdspj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:38:03.054688 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:03.054665    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdspj\" (UniqueName: \"kubernetes.io/projected/5e566e07-6079-435e-a406-7877c04168b8-kube-api-access-gdspj\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:03.054688 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:03.054687    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:03.054819 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:03.054698    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e566e07-6079-435e-a406-7877c04168b8-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:03.054819 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:03.054706    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e566e07-6079-435e-a406-7877c04168b8-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:03.145581 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:03.145559    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"]
Jul 09 18:38:03.148200 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:03.148183    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-prefill-78b7f4c847-97l8n"]
Jul 09 18:38:04.418019 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:04.417957    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e566e07-6079-435e-a406-7877c04168b8" path="/var/lib/kubelet/pods/5e566e07-6079-435e-a406-7877c04168b8/volumes"
Jul 09 18:38:12.221587 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.221531    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6" podUID="10c5aa67-b302-40b9-8b6c-27a0129d94bf" containerName="main" containerID="cri-o://f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586" gracePeriod=45
Jul 09 18:38:12.475221 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.475168    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:38:12.522277 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.522250    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-dshm\") pod \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") "
Jul 09 18:38:12.522422 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.522283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-home\") pod \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") "
Jul 09 18:38:12.522570 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.522549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-home" (OuterVolumeSpecName: "home") pod "10c5aa67-b302-40b9-8b6c-27a0129d94bf" (UID: "10c5aa67-b302-40b9-8b6c-27a0129d94bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:12.524245 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.524223    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-dshm" (OuterVolumeSpecName: "dshm") pod "10c5aa67-b302-40b9-8b6c-27a0129d94bf" (UID: "10c5aa67-b302-40b9-8b6c-27a0129d94bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:12.622736 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.622709    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tmp-dir\") pod \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") "
Jul 09 18:38:12.622917 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.622762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tls-certs\") pod \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") "
Jul 09 18:38:12.622917 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.622784    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-model-cache\") pod \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") "
Jul 09 18:38:12.622917 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.622830    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c5kb\" (UniqueName: \"kubernetes.io/projected/10c5aa67-b302-40b9-8b6c-27a0129d94bf-kube-api-access-9c5kb\") pod \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\" (UID: \"10c5aa67-b302-40b9-8b6c-27a0129d94bf\") "
Jul 09 18:38:12.623102 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.622991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10c5aa67-b302-40b9-8b6c-27a0129d94bf" (UID: "10c5aa67-b302-40b9-8b6c-27a0129d94bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:12.623102 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.623094    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:12.623206 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.623100    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "10c5aa67-b302-40b9-8b6c-27a0129d94bf" (UID: "10c5aa67-b302-40b9-8b6c-27a0129d94bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:38:12.623206 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.623112    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:12.623206 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.623133    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:12.624806 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.624781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10c5aa67-b302-40b9-8b6c-27a0129d94bf" (UID: "10c5aa67-b302-40b9-8b6c-27a0129d94bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:38:12.624914 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.624888    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c5aa67-b302-40b9-8b6c-27a0129d94bf-kube-api-access-9c5kb" (OuterVolumeSpecName: "kube-api-access-9c5kb") pod "10c5aa67-b302-40b9-8b6c-27a0129d94bf" (UID: "10c5aa67-b302-40b9-8b6c-27a0129d94bf"). InnerVolumeSpecName "kube-api-access-9c5kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:38:12.724316 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.724287    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10c5aa67-b302-40b9-8b6c-27a0129d94bf-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:12.724316 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.724311    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10c5aa67-b302-40b9-8b6c-27a0129d94bf-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:12.724316 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.724319    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c5kb\" (UniqueName: \"kubernetes.io/projected/10c5aa67-b302-40b9-8b6c-27a0129d94bf-kube-api-access-9c5kb\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:38:12.863360 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.863327    2577 generic.go:356] "Generic (PLEG): container finished" podID="10c5aa67-b302-40b9-8b6c-27a0129d94bf" containerID="f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586" exitCode=0
Jul 09 18:38:12.863524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.863371    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"
Jul 09 18:38:12.863524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.863411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6" event={"ID":"10c5aa67-b302-40b9-8b6c-27a0129d94bf","Type":"ContainerDied","Data":"f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586"}
Jul 09 18:38:12.863524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.863480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6" event={"ID":"10c5aa67-b302-40b9-8b6c-27a0129d94bf","Type":"ContainerDied","Data":"980e4835de02d5ab721b42b49b30032bc56636038775d65442b4701f91f04fb0"}
Jul 09 18:38:12.863524 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.863504    2577 scope.go:117] "RemoveContainer" containerID="f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586"
Jul 09 18:38:12.872058 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.872043    2577 scope.go:117] "RemoveContainer" containerID="f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586"
Jul 09 18:38:12.872293 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:38:12.872273    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586\": container with ID starting with f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586 not found: ID does not exist" containerID="f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586"
Jul 09 18:38:12.872335 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.872300    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586"} err="failed to get container status \"f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586\": rpc error: code = NotFound desc = could not find container \"f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586\": container with ID starting with f68999f9e45692f10a2c667352a9b01b71dcde4f5ca07b1362b0fadeeeec6586 not found: ID does not exist"
Jul 09 18:38:12.884739 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.884719    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"]
Jul 09 18:38:12.887910 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:12.887891    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-45st6"]
Jul 09 18:38:14.417494 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:14.417452    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10c5aa67-b302-40b9-8b6c-27a0129d94bf" path="/var/lib/kubelet/pods/10c5aa67-b302-40b9-8b6c-27a0129d94bf/volumes"
Jul 09 18:38:22.936488 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.936455    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"]
Jul 09 18:38:22.937038 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.936753    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c5aa67-b302-40b9-8b6c-27a0129d94bf" containerName="main"
Jul 09 18:38:22.937038 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.936765    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c5aa67-b302-40b9-8b6c-27a0129d94bf" containerName="main"
Jul 09 18:38:22.937038 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.936794    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e566e07-6079-435e-a406-7877c04168b8" containerName="main"
Jul 09 18:38:22.937038 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.936801    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e566e07-6079-435e-a406-7877c04168b8" containerName="main"
Jul 09 18:38:22.937038 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.936893    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c5aa67-b302-40b9-8b6c-27a0129d94bf" containerName="main"
Jul 09 18:38:22.937038 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.936911    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e566e07-6079-435e-a406-7877c04168b8" containerName="main"
Jul 09 18:38:22.941638 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.941606    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:22.945413 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.945381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-zk6cn\""
Jul 09 18:38:22.945413 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.945399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 09 18:38:22.947290 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.947267    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"]
Jul 09 18:38:22.950692 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.950675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:22.958962 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.958943    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"]
Jul 09 18:38:22.961952 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.961930    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"]
Jul 09 18:38:22.986615 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986578    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:22.986714 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:22.986714 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:22.986714 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986706    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:22.986889 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:22.986889 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:22.986889 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jmtg\" (UniqueName: \"kubernetes.io/projected/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-kube-api-access-8jmtg\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:22.986889 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:22.987083 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4fd\" (UniqueName: \"kubernetes.io/projected/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-kube-api-access-qr4fd\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:22.987083 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:22.987083 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.986972    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:22.987083 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:22.987015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088229 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.088229 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.088432 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.088432 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088276    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088432 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088298    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jmtg\" (UniqueName: \"kubernetes.io/projected/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-kube-api-access-8jmtg\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088432 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088324    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.088432 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088348    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4fd\" (UniqueName: \"kubernetes.io/projected/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-kube-api-access-qr4fd\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.088432 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088371    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088432 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.088799 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088799 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088799 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088799 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.088799 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.088799 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.089139 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.088984    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.089263 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.089240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.089496 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.089474    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.090746 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.090724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.091002 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.090947    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.091363 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.091343    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.091481 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.091458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.098310 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.098286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4fd\" (UniqueName: \"kubernetes.io/projected/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-kube-api-access-qr4fd\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.098396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.098292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jmtg\" (UniqueName: \"kubernetes.io/projected/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-kube-api-access-8jmtg\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.251341 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.251308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:23.260823 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.260796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:23.398736 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.398708    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"]
Jul 09 18:38:23.400607 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:38:23.400582    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90c1636_6a65_4ee5_8cc5_d53a8fb1be7e.slice/crio-d522b1e425ed1c36964bb2b125846a3827215d0ffb133585512d4585a886537e WatchSource:0}: Error finding container d522b1e425ed1c36964bb2b125846a3827215d0ffb133585512d4585a886537e: Status 404 returned error can't find the container with id d522b1e425ed1c36964bb2b125846a3827215d0ffb133585512d4585a886537e
Jul 09 18:38:23.402926 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.402909    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:38:23.417905 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.417881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"]
Jul 09 18:38:23.913368 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.913323    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd" event={"ID":"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd","Type":"ContainerStarted","Data":"a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8"}
Jul 09 18:38:23.913557 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.913405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd" event={"ID":"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd","Type":"ContainerStarted","Data":"ae43e543186da2b806da2bc4f574c354474f0fe821dd6b870ef14d7d6ed4f7ec"}
Jul 09 18:38:23.914546 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.914521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" event={"ID":"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e","Type":"ContainerStarted","Data":"d522b1e425ed1c36964bb2b125846a3827215d0ffb133585512d4585a886537e"}
Jul 09 18:38:23.938192 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:23.938146    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd" podStartSLOduration=1.938127271 podStartE2EDuration="1.938127271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:38:23.93639715 +0000 UTC m=+3538.126757067" watchObservedRunningTime="2026-07-09 18:38:23.938127271 +0000 UTC m=+3538.128487190"
Jul 09 18:38:24.260984 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:24.260961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:24.920536 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:24.920499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" event={"ID":"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e","Type":"ContainerStarted","Data":"d00c4216dbf466d8e00bb77fe4d363f4e2e8bb0eff880739707d74ecc6dacebf"}
Jul 09 18:38:24.920736 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:24.920698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:25.927344 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:25.927309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" event={"ID":"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e","Type":"ContainerStarted","Data":"7e57a0487e76172ccab6dbfddb0d2b8485eb7016a119e166599c557fef9e5c33"}
Jul 09 18:38:25.950352 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:25.950308    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" podStartSLOduration=3.178191487 podStartE2EDuration="3.950293568s" totalImagesPullingTime="772.102081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:38:25.94928354 +0000 UTC m=+3540.139643456" watchObservedRunningTime="2026-07-09 18:38:25.950293568 +0000 UTC m=+3540.140653482"
Jul 09 18:38:26.251857 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:26.251811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:33.251945 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:33.251912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:33.261597 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:33.261569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:33.269527 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:33.269504    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:33.274004 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:33.273985    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:33.972700 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:33.972677    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:33.974237 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:33.974212    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:38:34.979913 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:34.979885    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:38:56.831336 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:56.831297    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"]
Jul 09 18:38:56.835289 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:38:56.835260    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"]
Jul 09 18:39:11.851876 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:11.851805    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd" podUID="9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" containerName="main" containerID="cri-o://a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8" gracePeriod=45
Jul 09 18:39:11.855855 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:11.855805    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="main" containerID="cri-o://7e57a0487e76172ccab6dbfddb0d2b8485eb7016a119e166599c557fef9e5c33" gracePeriod=45
Jul 09 18:39:11.962443 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:11.962408    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd" podUID="9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 09 18:39:11.962560 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:11.962439    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 09 18:39:11.972642 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:11.972616    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="llm-d-routing-sidecar" containerID="cri-o://d00c4216dbf466d8e00bb77fe4d363f4e2e8bb0eff880739707d74ecc6dacebf" gracePeriod=45
Jul 09 18:39:12.109186 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.109135    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:39:12.130743 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.130663    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" containerID="a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8" exitCode=0
Jul 09 18:39:12.131162 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.131105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd" event={"ID":"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd","Type":"ContainerDied","Data":"a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8"}
Jul 09 18:39:12.131651 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.131593    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"
Jul 09 18:39:12.132572 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.131789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd" event={"ID":"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd","Type":"ContainerDied","Data":"ae43e543186da2b806da2bc4f574c354474f0fe821dd6b870ef14d7d6ed4f7ec"}
Jul 09 18:39:12.132572 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.131870    2577 scope.go:117] "RemoveContainer" containerID="a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8"
Jul 09 18:39:12.133951 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.133928    2577 generic.go:356] "Generic (PLEG): container finished" podID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerID="7e57a0487e76172ccab6dbfddb0d2b8485eb7016a119e166599c557fef9e5c33" exitCode=0
Jul 09 18:39:12.134053 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.133983    2577 generic.go:356] "Generic (PLEG): container finished" podID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerID="d00c4216dbf466d8e00bb77fe4d363f4e2e8bb0eff880739707d74ecc6dacebf" exitCode=0
Jul 09 18:39:12.134192 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.134132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" event={"ID":"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e","Type":"ContainerDied","Data":"7e57a0487e76172ccab6dbfddb0d2b8485eb7016a119e166599c557fef9e5c33"}
Jul 09 18:39:12.134278 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.134179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" event={"ID":"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e","Type":"ContainerDied","Data":"d00c4216dbf466d8e00bb77fe4d363f4e2e8bb0eff880739707d74ecc6dacebf"}
Jul 09 18:39:12.142005 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.141986    2577 scope.go:117] "RemoveContainer" containerID="a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8"
Jul 09 18:39:12.142327 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:39:12.142302    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8\": container with ID starting with a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8 not found: ID does not exist" containerID="a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8"
Jul 09 18:39:12.142416 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.142336    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8"} err="failed to get container status \"a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8\": rpc error: code = NotFound desc = could not find container \"a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8\": container with ID starting with a6d7672ed0e287ce981c38e8a06d3638201df749678116d15bd387b90bf2a4a8 not found: ID does not exist"
Jul 09 18:39:12.179097 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179076    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tmp-dir\") pod \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") "
Jul 09 18:39:12.179237 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179123    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-dshm\") pod \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") "
Jul 09 18:39:12.179237 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179162    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tls-certs\") pod \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") "
Jul 09 18:39:12.179237 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179187    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-model-cache\") pod \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") "
Jul 09 18:39:12.179418 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179239    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-home\") pod \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") "
Jul 09 18:39:12.179418 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179262    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jmtg\" (UniqueName: \"kubernetes.io/projected/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-kube-api-access-8jmtg\") pod \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\" (UID: \"9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd\") "
Jul 09 18:39:12.179418 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179364    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" (UID: "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.179571 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179449    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.179627 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179571    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-home" (OuterVolumeSpecName: "home") pod "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" (UID: "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.179885 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.179792    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" (UID: "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.181692 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.181670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" (UID: "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:39:12.181817 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.181782    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-dshm" (OuterVolumeSpecName: "dshm") pod "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" (UID: "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.181946 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.181810    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-kube-api-access-8jmtg" (OuterVolumeSpecName: "kube-api-access-8jmtg") pod "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" (UID: "9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd"). InnerVolumeSpecName "kube-api-access-8jmtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:39:12.219193 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.219174    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:39:12.280624 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.280589    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.280624 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.280624    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.280855 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.280641    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.280855 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.280655    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.280855 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.280669    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jmtg\" (UniqueName: \"kubernetes.io/projected/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd-kube-api-access-8jmtg\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.381703 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381617    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-model-cache\") pod \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") "
Jul 09 18:39:12.381703 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381659    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tmp-dir\") pod \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") "
Jul 09 18:39:12.381703 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381702    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-dshm\") pod \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") "
Jul 09 18:39:12.382026 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381725    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr4fd\" (UniqueName: \"kubernetes.io/projected/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-kube-api-access-qr4fd\") pod \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") "
Jul 09 18:39:12.382026 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tls-certs\") pod \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") "
Jul 09 18:39:12.382026 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381798    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-home\") pod \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\" (UID: \"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e\") "
Jul 09 18:39:12.382026 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381955    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" (UID: "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.382026 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.381985    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" (UID: "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.382317 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.382064    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-model-cache\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.382317 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.382084    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tmp-dir\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.382317 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.382281    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-home" (OuterVolumeSpecName: "home") pod "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" (UID: "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.383998 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.383973    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-kube-api-access-qr4fd" (OuterVolumeSpecName: "kube-api-access-qr4fd") pod "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" (UID: "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e"). InnerVolumeSpecName "kube-api-access-qr4fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:39:12.384293 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.384265    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-dshm" (OuterVolumeSpecName: "dshm") pod "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" (UID: "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:39:12.384396 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.384342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" (UID: "a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:39:12.450559 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.450535    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"]
Jul 09 18:39:12.455022 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.455002    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc5j9qhd"]
Jul 09 18:39:12.482499 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.482481    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-dshm\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.482499 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.482499    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr4fd\" (UniqueName: \"kubernetes.io/projected/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-kube-api-access-qr4fd\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.482624 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.482509    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-tls-certs\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:12.482624 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:12.482519    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e-home\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:13.140704 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:13.140629    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"
Jul 09 18:39:13.140704 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:13.140648    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk" event={"ID":"a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e","Type":"ContainerDied","Data":"d522b1e425ed1c36964bb2b125846a3827215d0ffb133585512d4585a886537e"}
Jul 09 18:39:13.141236 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:13.140707    2577 scope.go:117] "RemoveContainer" containerID="7e57a0487e76172ccab6dbfddb0d2b8485eb7016a119e166599c557fef9e5c33"
Jul 09 18:39:13.148728 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:13.148705    2577 scope.go:117] "RemoveContainer" containerID="d00c4216dbf466d8e00bb77fe4d363f4e2e8bb0eff880739707d74ecc6dacebf"
Jul 09 18:39:13.159925 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:13.159900    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"]
Jul 09 18:39:13.164174 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:13.164146    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-4l7qk"]
Jul 09 18:39:14.417160 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:14.417129    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" path="/var/lib/kubelet/pods/9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd/volumes"
Jul 09 18:39:14.417530 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:14.417495    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" path="/var/lib/kubelet/pods/a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e/volumes"
Jul 09 18:39:23.470554 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.470516    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7787c55b6-zt6xb"]
Jul 09 18:39:23.471030 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.470677    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb" podUID="879849b1-cf75-46cd-9086-79a538a95ead" containerName="manager" containerID="cri-o://06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db" gracePeriod=30
Jul 09 18:39:23.721208 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.721145    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 18:39:23.871064 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.871033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert\") pod \"879849b1-cf75-46cd-9086-79a538a95ead\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") "
Jul 09 18:39:23.871064 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.871072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp2wh\" (UniqueName: \"kubernetes.io/projected/879849b1-cf75-46cd-9086-79a538a95ead-kube-api-access-xp2wh\") pod \"879849b1-cf75-46cd-9086-79a538a95ead\" (UID: \"879849b1-cf75-46cd-9086-79a538a95ead\") "
Jul 09 18:39:23.873167 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.873135    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert" (OuterVolumeSpecName: "cert") pod "879849b1-cf75-46cd-9086-79a538a95ead" (UID: "879849b1-cf75-46cd-9086-79a538a95ead"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:39:23.873287 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.873172    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/879849b1-cf75-46cd-9086-79a538a95ead-kube-api-access-xp2wh" (OuterVolumeSpecName: "kube-api-access-xp2wh") pod "879849b1-cf75-46cd-9086-79a538a95ead" (UID: "879849b1-cf75-46cd-9086-79a538a95ead"). InnerVolumeSpecName "kube-api-access-xp2wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:39:23.972605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.972555    2577 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/879849b1-cf75-46cd-9086-79a538a95ead-cert\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:23.972605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:23.972580    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp2wh\" (UniqueName: \"kubernetes.io/projected/879849b1-cf75-46cd-9086-79a538a95ead-kube-api-access-xp2wh\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:39:24.193646 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.193604    2577 generic.go:356] "Generic (PLEG): container finished" podID="879849b1-cf75-46cd-9086-79a538a95ead" containerID="06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db" exitCode=0
Jul 09 18:39:24.193878 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.193650    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb"
Jul 09 18:39:24.193878 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.193689    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb" event={"ID":"879849b1-cf75-46cd-9086-79a538a95ead","Type":"ContainerDied","Data":"06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db"}
Jul 09 18:39:24.193878 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.193775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7787c55b6-zt6xb" event={"ID":"879849b1-cf75-46cd-9086-79a538a95ead","Type":"ContainerDied","Data":"d7f7a06994eb70c916c66e7f78bd2f1662841bc68e28d124df578c8d28afdcc2"}
Jul 09 18:39:24.193878 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.193802    2577 scope.go:117] "RemoveContainer" containerID="06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db"
Jul 09 18:39:24.204821 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.204801    2577 scope.go:117] "RemoveContainer" containerID="06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db"
Jul 09 18:39:24.205110 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:39:24.205088    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db\": container with ID starting with 06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db not found: ID does not exist" containerID="06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db"
Jul 09 18:39:24.205186 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.205118    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db"} err="failed to get container status \"06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db\": rpc error: code = NotFound desc = could not find container \"06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db\": container with ID starting with 06bca615e4f626dafd5ebe352bc02d612246f45668d2bc11e884d010f35662db not found: ID does not exist"
Jul 09 18:39:24.216301 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.216271    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-7787c55b6-zt6xb"]
Jul 09 18:39:24.218305 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.218276    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-7787c55b6-zt6xb"]
Jul 09 18:39:24.417216 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:24.417182    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="879849b1-cf75-46cd-9086-79a538a95ead" path="/var/lib/kubelet/pods/879849b1-cf75-46cd-9086-79a538a95ead/volumes"
Jul 09 18:39:26.492982 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:26.492875    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:39:26.518569 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:39:26.501003    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sh5ck_8abf9808-a4c1-4b03-8952-af5a3f1b63a3/ovn-acl-logging/0.log"
Jul 09 18:40:01.483962 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.483927    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2kvkt/must-gather-cg9r9"]
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484212    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="879849b1-cf75-46cd-9086-79a538a95ead" containerName="manager"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484224    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="879849b1-cf75-46cd-9086-79a538a95ead" containerName="manager"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484233    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" containerName="main"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484238    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" containerName="main"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484249    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="main"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484255    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="main"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484266    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="llm-d-routing-sidecar"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484271    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="llm-d-routing-sidecar"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484323    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="main"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484332    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="879849b1-cf75-46cd-9086-79a538a95ead" containerName="manager"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484339    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a90c1636-6a65-4ee5-8cc5-d53a8fb1be7e" containerName="llm-d-routing-sidecar"
Jul 09 18:40:01.484397 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.484347    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c4e8b2d-ce0b-4f00-b0c7-68723985c7dd" containerName="main"
Jul 09 18:40:01.487326 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.487306    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.490059 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.490028    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2kvkt\"/\"kube-root-ca.crt\""
Jul 09 18:40:01.490468 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.490451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2kvkt\"/\"openshift-service-ca.crt\""
Jul 09 18:40:01.497303 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.497280    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kvkt/must-gather-cg9r9"]
Jul 09 18:40:01.536605 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.536582    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktm47\" (UniqueName: \"kubernetes.io/projected/ca62180b-740f-4b8d-b834-2029d2bb17fc-kube-api-access-ktm47\") pod \"must-gather-cg9r9\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") " pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.536710 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.536627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca62180b-740f-4b8d-b834-2029d2bb17fc-must-gather-output\") pod \"must-gather-cg9r9\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") " pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.637929 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.637904    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca62180b-740f-4b8d-b834-2029d2bb17fc-must-gather-output\") pod \"must-gather-cg9r9\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") " pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.638042 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.637963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktm47\" (UniqueName: \"kubernetes.io/projected/ca62180b-740f-4b8d-b834-2029d2bb17fc-kube-api-access-ktm47\") pod \"must-gather-cg9r9\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") " pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.638248 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.638228    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca62180b-740f-4b8d-b834-2029d2bb17fc-must-gather-output\") pod \"must-gather-cg9r9\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") " pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.648725 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.648702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktm47\" (UniqueName: \"kubernetes.io/projected/ca62180b-740f-4b8d-b834-2029d2bb17fc-kube-api-access-ktm47\") pod \"must-gather-cg9r9\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") " pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.796400 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.796373    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:40:01.929185 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:01.929154    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2kvkt/must-gather-cg9r9"]
Jul 09 18:40:01.932112 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:40:01.932085    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca62180b_740f_4b8d_b834_2029d2bb17fc.slice/crio-274aa97ffe9760219ea73b6fd0450283b166231e175763c5818a12809bc62449 WatchSource:0}: Error finding container 274aa97ffe9760219ea73b6fd0450283b166231e175763c5818a12809bc62449: Status 404 returned error can't find the container with id 274aa97ffe9760219ea73b6fd0450283b166231e175763c5818a12809bc62449
Jul 09 18:40:02.354111 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:02.354078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kvkt/must-gather-cg9r9" event={"ID":"ca62180b-740f-4b8d-b834-2029d2bb17fc","Type":"ContainerStarted","Data":"274aa97ffe9760219ea73b6fd0450283b166231e175763c5818a12809bc62449"}
Jul 09 18:40:09.401451 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:09.401409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kvkt/must-gather-cg9r9" event={"ID":"ca62180b-740f-4b8d-b834-2029d2bb17fc","Type":"ContainerStarted","Data":"f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a"}
Jul 09 18:40:09.401953 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:09.401459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kvkt/must-gather-cg9r9" event={"ID":"ca62180b-740f-4b8d-b834-2029d2bb17fc","Type":"ContainerStarted","Data":"de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5"}
Jul 09 18:40:09.420825 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:09.420772    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2kvkt/must-gather-cg9r9" podStartSLOduration=1.64330954 podStartE2EDuration="8.420756025s" totalImagesPullingTime="6.777446485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:40:09.41990164 +0000 UTC m=+3643.610261557" watchObservedRunningTime="2026-07-09 18:40:09.420756025 +0000 UTC m=+3643.611115939"
Jul 09 18:40:18.163697 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:18.163670    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:18.270023 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:18.269987    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:18.280224 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:18.280198    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:19.879218 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:19.879187    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:19.965560 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:19.965523    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:19.976034 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:19.976009    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:21.554410 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:21.554368    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:21.633393 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:21.633369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:21.650701 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:21.650666    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:23.159454 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:23.159424    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:23.242654 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:23.242628    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:23.252106 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:23.252083    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:24.897118 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:24.897080    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:24.975851 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:24.975816    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:24.987066 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:24.987044    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:26.506392 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:26.506360    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:26.593368 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:26.593339    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:26.614814 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:26.614790    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:28.212576 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:28.212539    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:28.307734 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:28.307696    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:28.329890 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:28.329862    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:29.947505 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:29.947476    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:30.039568 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:30.039537    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:30.050457 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:30.050432    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:31.626971 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:31.626934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:31.703047 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:31.703015    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:31.718463 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:31.718442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:33.300248 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:33.300219    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:33.376423 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:33.376399    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:33.387763 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:33.387738    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:34.929176 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:34.929134    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:35.004474 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:35.004448    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:35.017663 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:35.017642    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:36.592550 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:36.592517    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:36.672722 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:36.672695    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:36.685754 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:36.685725    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:38.206757 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:38.206710    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:38.298403 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:38.298341    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:38.309360 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:38.309335    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:39.750724 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:39.750677    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:39.854964 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:39.854934    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:39.866076 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:39.866053    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:41.403174 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:41.403148    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:41.481789 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:41.481760    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:41.494642 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:41.494607    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:43.009771 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:43.009738    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-q2skc_6f2f5fd1-2d8a-4656-b792-ed15ed54539c/storage-initializer/0.log"
Jul 09 18:40:43.085649 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:43.085624    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/main/0.log"
Jul 09 18:40:43.095816 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:43.095797    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schesxccr_6c5e7685-a76b-42fa-8048-fb682f709bfe/storage-initializer/0.log"
Jul 09 18:40:45.441562 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:45.441535    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/0.log"
Jul 09 18:40:45.509721 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:45.509683    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/1.log"
Jul 09 18:40:47.323642 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:47.323603    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/0.log"
Jul 09 18:40:47.388190 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:47.388162    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/1.log"
Jul 09 18:40:49.236161 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:49.236063    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/0.log"
Jul 09 18:40:49.299270 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:49.299243    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/1.log"
Jul 09 18:40:51.095494 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:51.095463    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/0.log"
Jul 09 18:40:51.156884 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:40:51.156857    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/1.log"
Jul 09 18:41:07.683487 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:07.683457    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerID="de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5" exitCode=0
Jul 09 18:41:07.683874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:07.683531    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2kvkt/must-gather-cg9r9" event={"ID":"ca62180b-740f-4b8d-b834-2029d2bb17fc","Type":"ContainerDied","Data":"de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5"}
Jul 09 18:41:07.683874 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:07.683857    2577 scope.go:117] "RemoveContainer" containerID="de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5"
Jul 09 18:41:08.464031 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:08.463998    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2kvkt_must-gather-cg9r9_ca62180b-740f-4b8d-b834-2029d2bb17fc/gather/0.log"
Jul 09 18:41:09.308478 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.308445    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xlxbw/must-gather-5lnvf"]
Jul 09 18:41:09.312986 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.312967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.315650 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.315632    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlxbw\"/\"kube-root-ca.crt\""
Jul 09 18:41:09.316864 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.316831    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xlxbw\"/\"default-dockercfg-wbfr7\""
Jul 09 18:41:09.316943 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.316831    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xlxbw\"/\"openshift-service-ca.crt\""
Jul 09 18:41:09.319947 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.319926    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlxbw/must-gather-5lnvf"]
Jul 09 18:41:09.401260 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.401230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4049e48d-5c86-4c5b-9345-24244a048be4-must-gather-output\") pod \"must-gather-5lnvf\" (UID: \"4049e48d-5c86-4c5b-9345-24244a048be4\") " pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.401362 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.401264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvkk\" (UniqueName: \"kubernetes.io/projected/4049e48d-5c86-4c5b-9345-24244a048be4-kube-api-access-4qvkk\") pod \"must-gather-5lnvf\" (UID: \"4049e48d-5c86-4c5b-9345-24244a048be4\") " pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.502320 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.502291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4049e48d-5c86-4c5b-9345-24244a048be4-must-gather-output\") pod \"must-gather-5lnvf\" (UID: \"4049e48d-5c86-4c5b-9345-24244a048be4\") " pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.502425 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.502328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvkk\" (UniqueName: \"kubernetes.io/projected/4049e48d-5c86-4c5b-9345-24244a048be4-kube-api-access-4qvkk\") pod \"must-gather-5lnvf\" (UID: \"4049e48d-5c86-4c5b-9345-24244a048be4\") " pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.502626 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.502608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4049e48d-5c86-4c5b-9345-24244a048be4-must-gather-output\") pod \"must-gather-5lnvf\" (UID: \"4049e48d-5c86-4c5b-9345-24244a048be4\") " pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.521891 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.521866    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvkk\" (UniqueName: \"kubernetes.io/projected/4049e48d-5c86-4c5b-9345-24244a048be4-kube-api-access-4qvkk\") pod \"must-gather-5lnvf\" (UID: \"4049e48d-5c86-4c5b-9345-24244a048be4\") " pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.622622 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.622531    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlxbw/must-gather-5lnvf"
Jul 09 18:41:09.754725 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:09.754600    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlxbw/must-gather-5lnvf"]
Jul 09 18:41:09.757128 ip-10-0-141-44 kubenswrapper[2577]: W0709 18:41:09.757098    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4049e48d_5c86_4c5b_9345_24244a048be4.slice/crio-415d29dec592c4d4c347e68b24136291c669b84c6f8aa804f136eee64f61bb54 WatchSource:0}: Error finding container 415d29dec592c4d4c347e68b24136291c669b84c6f8aa804f136eee64f61bb54: Status 404 returned error can't find the container with id 415d29dec592c4d4c347e68b24136291c669b84c6f8aa804f136eee64f61bb54
Jul 09 18:41:10.698891 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:10.698853    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlxbw/must-gather-5lnvf" event={"ID":"4049e48d-5c86-4c5b-9345-24244a048be4","Type":"ContainerStarted","Data":"415d29dec592c4d4c347e68b24136291c669b84c6f8aa804f136eee64f61bb54"}
Jul 09 18:41:11.706855 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:11.706794    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlxbw/must-gather-5lnvf" event={"ID":"4049e48d-5c86-4c5b-9345-24244a048be4","Type":"ContainerStarted","Data":"3a63db83ba7a10e74c6a3c9280251de12e3001ef77eabe4f4c68ae874a84ea97"}
Jul 09 18:41:11.707329 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:11.706870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlxbw/must-gather-5lnvf" event={"ID":"4049e48d-5c86-4c5b-9345-24244a048be4","Type":"ContainerStarted","Data":"4ea74aa2257ff1e85ad11f79445ab134cf63204e18913a2766810dca15decef5"}
Jul 09 18:41:12.388575 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:12.388551    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-s45x2_9af1998d-6f11-4d4c-b45c-bcf73f88e1a1/global-pull-secret-syncer/0.log"
Jul 09 18:41:12.506530 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:12.506500    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lcjp2_de5fad1c-2d42-4610-97f3-595c50f0158e/konnectivity-agent/0.log"
Jul 09 18:41:12.578937 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:12.578907    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-44.ec2.internal_89b829e494294382f2ba3cb0e5e2f900/haproxy/0.log"
Jul 09 18:41:14.156295 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.156233    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xlxbw/must-gather-5lnvf" podStartSLOduration=4.17410311 podStartE2EDuration="5.156206978s" totalImagesPullingTime="982.103868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:41:11.723299815 +0000 UTC m=+3705.913659732" watchObservedRunningTime="2026-07-09 18:41:14.156206978 +0000 UTC m=+3708.346566894"
Jul 09 18:41:14.158723 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.158693    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2kvkt/must-gather-cg9r9"]
Jul 09 18:41:14.159110 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.159083    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-2kvkt/must-gather-cg9r9" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerName="copy" containerID="cri-o://f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a" gracePeriod=2
Jul 09 18:41:14.165950 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.165924    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2kvkt/must-gather-cg9r9"]
Jul 09 18:41:14.530865 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.530196    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2kvkt_must-gather-cg9r9_ca62180b-740f-4b8d-b834-2029d2bb17fc/copy/0.log"
Jul 09 18:41:14.534857 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.533062    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:41:14.650134 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.650096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktm47\" (UniqueName: \"kubernetes.io/projected/ca62180b-740f-4b8d-b834-2029d2bb17fc-kube-api-access-ktm47\") pod \"ca62180b-740f-4b8d-b834-2029d2bb17fc\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") "
Jul 09 18:41:14.650329 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.650158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca62180b-740f-4b8d-b834-2029d2bb17fc-must-gather-output\") pod \"ca62180b-740f-4b8d-b834-2029d2bb17fc\" (UID: \"ca62180b-740f-4b8d-b834-2029d2bb17fc\") "
Jul 09 18:41:14.654139 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.654106    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca62180b-740f-4b8d-b834-2029d2bb17fc-kube-api-access-ktm47" (OuterVolumeSpecName: "kube-api-access-ktm47") pod "ca62180b-740f-4b8d-b834-2029d2bb17fc" (UID: "ca62180b-740f-4b8d-b834-2029d2bb17fc"). InnerVolumeSpecName "kube-api-access-ktm47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:41:14.665364 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.665320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca62180b-740f-4b8d-b834-2029d2bb17fc-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "ca62180b-740f-4b8d-b834-2029d2bb17fc" (UID: "ca62180b-740f-4b8d-b834-2029d2bb17fc"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:41:14.725625 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.725589    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2kvkt_must-gather-cg9r9_ca62180b-740f-4b8d-b834-2029d2bb17fc/copy/0.log"
Jul 09 18:41:14.726071 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.726040    2577 generic.go:356] "Generic (PLEG): container finished" podID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerID="f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a" exitCode=143
Jul 09 18:41:14.726201 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.726151    2577 scope.go:117] "RemoveContainer" containerID="f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a"
Jul 09 18:41:14.726329 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.726313    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2kvkt/must-gather-cg9r9"
Jul 09 18:41:14.749957 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.749672    2577 scope.go:117] "RemoveContainer" containerID="de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5"
Jul 09 18:41:14.751205 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.750802    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktm47\" (UniqueName: \"kubernetes.io/projected/ca62180b-740f-4b8d-b834-2029d2bb17fc-kube-api-access-ktm47\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:41:14.751205 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.750826    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca62180b-740f-4b8d-b834-2029d2bb17fc-must-gather-output\") on node \"ip-10-0-141-44.ec2.internal\" DevicePath \"\""
Jul 09 18:41:14.775475 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.775447    2577 scope.go:117] "RemoveContainer" containerID="f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a"
Jul 09 18:41:14.775920 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:41:14.775886    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a\": container with ID starting with f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a not found: ID does not exist" containerID="f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a"
Jul 09 18:41:14.776021 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.775931    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a"} err="failed to get container status \"f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a\": rpc error: code = NotFound desc = could not find container \"f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a\": container with ID starting with f653e422eb375830434611b6e27ea84b97133c218955f5ab55b2e4fa09c55c6a not found: ID does not exist"
Jul 09 18:41:14.776021 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.775956    2577 scope.go:117] "RemoveContainer" containerID="de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5"
Jul 09 18:41:14.776274 ip-10-0-141-44 kubenswrapper[2577]: E0709 18:41:14.776248    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5\": container with ID starting with de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5 not found: ID does not exist" containerID="de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5"
Jul 09 18:41:14.776358 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:14.776282    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5"} err="failed to get container status \"de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5\": rpc error: code = NotFound desc = could not find container \"de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5\": container with ID starting with de3f43912af7a3cfbc91b232111da45e7e6b91a53e296ac9f7f6f471efe8f5e5 not found: ID does not exist"
Jul 09 18:41:16.425149 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:16.425112    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" path="/var/lib/kubelet/pods/ca62180b-740f-4b8d-b834-2029d2bb17fc/volumes"
Jul 09 18:41:20.789985 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:20.789928    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wl64d_b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a/node-exporter/0.log"
Jul 09 18:41:20.811811 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:20.811775    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wl64d_b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a/kube-rbac-proxy/0.log"
Jul 09 18:41:20.836868 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:20.836820    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wl64d_b2358ae9-30ad-4bc9-b8da-d1c31d7bd70a/init-textfile/0.log"
Jul 09 18:41:23.387876 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.387829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"]
Jul 09 18:41:23.388637 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.388615    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerName="gather"
Jul 09 18:41:23.388637 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.388637    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerName="gather"
Jul 09 18:41:23.388800 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.388665    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerName="copy"
Jul 09 18:41:23.388800 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.388673    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerName="copy"
Jul 09 18:41:23.388800 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.388751    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerName="gather"
Jul 09 18:41:23.388800 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.388766    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca62180b-740f-4b8d-b834-2029d2bb17fc" containerName="copy"
Jul 09 18:41:23.397381 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.397350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.402996 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.402935    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"]
Jul 09 18:41:23.443212 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.443173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-proc\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.443462 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.443443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-sys\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.443624 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.443605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-lib-modules\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.443759 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.443742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctdvd\" (UniqueName: \"kubernetes.io/projected/bbf43902-d352-4540-a701-ab4a1ce851fb-kube-api-access-ctdvd\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.443920 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.443903    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-podres\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.545314 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.545281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-podres\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.545578 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.545560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-proc\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.546420 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.545951    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-sys\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.546420 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.546029    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-sys\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.546420 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.545709    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-proc\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.546420 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.546057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-lib-modules\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.546420 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.546095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctdvd\" (UniqueName: \"kubernetes.io/projected/bbf43902-d352-4540-a701-ab4a1ce851fb-kube-api-access-ctdvd\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.546420 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.545470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-podres\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.546753 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.546512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bbf43902-d352-4540-a701-ab4a1ce851fb-lib-modules\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.554085 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.554059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctdvd\" (UniqueName: \"kubernetes.io/projected/bbf43902-d352-4540-a701-ab4a1ce851fb-kube-api-access-ctdvd\") pod \"perf-node-gather-daemonset-mfkg6\" (UID: \"bbf43902-d352-4540-a701-ab4a1ce851fb\") " pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.711061 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.710976    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:23.908087 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:23.906332    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"]
Jul 09 18:41:24.779822 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:24.779751    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-255cj_23ef0c45-984c-42cd-b9b1-761ade3987f9/volume-data-source-validator/0.log"
Jul 09 18:41:24.791522 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:24.791439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6" event={"ID":"bbf43902-d352-4540-a701-ab4a1ce851fb","Type":"ContainerStarted","Data":"51cc7354c5dbeaed419662c950cd14f5a4a26937910b1737c8f94696bebab934"}
Jul 09 18:41:24.791522 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:24.791502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6" event={"ID":"bbf43902-d352-4540-a701-ab4a1ce851fb","Type":"ContainerStarted","Data":"ac7cd87317af48743f386e3b9e833414d2c813227a61e01f061bca1a6b50f9c5"}
Jul 09 18:41:24.791873 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:24.791534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:24.813114 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:24.813051    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6" podStartSLOduration=1.813029707 podStartE2EDuration="1.813029707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:41:24.81129573 +0000 UTC m=+3719.001655647" watchObservedRunningTime="2026-07-09 18:41:24.813029707 +0000 UTC m=+3719.003389624"
Jul 09 18:41:25.870723 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:25.870695    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82fgg_fa624b20-2ac3-494f-b333-36a45b55a991/dns/0.log"
Jul 09 18:41:25.892504 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:25.892468    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-82fgg_fa624b20-2ac3-494f-b333-36a45b55a991/kube-rbac-proxy/0.log"
Jul 09 18:41:26.027419 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:26.027396    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8j9n9_25c30946-998e-4d0d-8673-a898a8247bd7/dns-node-resolver/0.log"
Jul 09 18:41:26.707203 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:26.707124    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m9ntl_338f5cc2-493a-43df-8aa6-a1e4b609de1f/node-ca/0.log"
Jul 09 18:41:28.602066 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:28.602020    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/0.log"
Jul 09 18:41:28.681485 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:28.681444    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/1.log"
Jul 09 18:41:29.279360 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:29.279332    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6cwhv_f04e2cc9-36ef-482b-a971-b66cb8cfd2a5/serve-healthcheck-canary/0.log"
Jul 09 18:41:29.862892 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:29.862858    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vv52_61d30fe0-6f1d-467e-9e00-bc5d16ecb399/kube-rbac-proxy/0.log"
Jul 09 18:41:29.887711 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:29.887688    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vv52_61d30fe0-6f1d-467e-9e00-bc5d16ecb399/exporter/0.log"
Jul 09 18:41:29.910819 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:29.910788    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5vv52_61d30fe0-6f1d-467e-9e00-bc5d16ecb399/extractor/0.log"
Jul 09 18:41:30.810555 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:30.810529    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xlxbw/perf-node-gather-daemonset-mfkg6"
Jul 09 18:41:33.329224 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:33.329190    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6cd758bd69-zwxvw_50aef266-a676-4053-9bcc-ffb04572b455/manager/0.log"
Jul 09 18:41:33.356906 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:33.356870    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tlsc4_da0085d4-4998-4cfa-970c-265c787d1e74/openshift-lws-operator/0.log"
Jul 09 18:41:34.259328 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:34.259289    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-vl2hg_80a8c86d-d109-490e-abe9-9e4d22a3caf5/server/0.log"
Jul 09 18:41:35.245479 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:35.245452    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-77l6h_2c4791d2-4d43-4b09-8688-49742c15025b/manager/0.log"
Jul 09 18:41:35.284223 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:35.284177    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5n4x5_bdc3e083-903b-4854-9a3a-53798fe56051/s3-init/0.log"
Jul 09 18:41:35.309295 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:35.309276    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-5n4x5_bdc3e083-903b-4854-9a3a-53798fe56051/download-hf-model/0.log"
Jul 09 18:41:36.035856 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:36.035811    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/0.log"
Jul 09 18:41:36.127568 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:36.127502    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/1.log"
Jul 09 18:41:38.047660 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:38.047626    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/0.log"
Jul 09 18:41:38.128674 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:38.128619    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-cvzqk_cf762c13-445f-43bd-912e-096388217608/istio-proxy/1.log"
Jul 09 18:41:40.813377 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:40.813349    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-rxzlx_42f5ac83-fa33-43c3-a27c-7e5c195bf93b/migrator/0.log"
Jul 09 18:41:40.834608 ip-10-0-141-44 kubenswrapper[2577]: I0709 18:41:40.834582    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-rxzlx_42f5ac83-fa33-43c3-a27c-7e5c195bf93b/graceful-termination/0.log"