Sep 18 16:13:54.046284 ip-10-0-130-101 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 16:13:54.046297 ip-10-0-130-101 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 16:13:54.046307 ip-10-0-130-101 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 16:13:54.046534 ip-10-0-130-101 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 16:14:04.158163 ip-10-0-130-101 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 18 16:14:04.158180 ip-10-0-130-101 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 37222339bab348939b3299d1cbf667a1 --
Sep 18 16:16:04.666560 ip-10-0-130-101 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 16:16:08.112505 ip-10-0-130-101 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:08.112505 ip-10-0-130-101 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 16:16:08.113715 ip-10-0-130-101 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:08.113715 ip-10-0-130-101 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 16:16:08.113715 ip-10-0-130-101 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 16:16:08.114054 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.113696    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 16:16:08.116898 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116881    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:08.116898 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116898    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:08.116898 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116902    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116906    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116909    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116914    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116918    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116921    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116924    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116927    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116930    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116933    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116935    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116938    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116940    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116943    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116946    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116948    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116951    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116953    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116956    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:08.117027 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116959    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116961    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116965    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116973    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116976    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116979    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116982    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116985    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116988    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116990    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116993    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116995    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.116998    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117000    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117003    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117006    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117009    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117011    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117014    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117017    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:08.117515 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117020    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117022    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117025    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117028    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117030    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117033    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117036    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117040    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117043    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117045    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117047    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117050    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117053    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117055    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117058    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117061    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117064    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117066    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117069    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117071    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:08.118054 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117075    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117077    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117080    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117082    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117085    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117088    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117092    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117097    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117100    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117103    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117106    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117108    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117111    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117113    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117116    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117119    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117122    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117124    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117127    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:08.118551 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117129    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117132    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117135    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117139    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117142    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117145    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117147    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117150    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117152    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117155    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117157    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117160    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117162    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117165    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117167    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117170    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117173    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117175    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117178    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117890    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:08.119014 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117898    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117901    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117904    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117907    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117910    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117913    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117916    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117919    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117921    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117924    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117927    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117929    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117932    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117935    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117938    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117941    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117943    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117946    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117949    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117952    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:08.119561 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117954    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117957    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117959    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117962    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117964    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117967    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117969    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117972    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117974    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117977    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117980    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117984    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117989    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117994    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.117997    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118001    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118003    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118006    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118009    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:08.120079 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118011    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118014    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118016    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118019    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118022    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118024    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118027    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118030    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118033    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118035    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118038    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118040    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118043    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118047    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118051    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118053    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118056    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118059    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118063    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:08.120573 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118066    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118069    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118073    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118075    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118078    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118081    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118086    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118089    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118092    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118095    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118098    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118101    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118103    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118106    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118108    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118111    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118114    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118116    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118119    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118121    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:08.121044 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118124    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118127    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118130    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118133    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118135    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118138    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118140    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118143    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118145    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118148    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118151    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118153    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118156    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118158    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118161    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118163    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118166    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118168    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118172    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:08.121555 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118174    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118602    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118607    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118611    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118614    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118616    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118619    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118621    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118624    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118627    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118630    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118632    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118635    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118637    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118640    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118643    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118646    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118648    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118651    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118654    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:08.122021 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118656    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118659    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118662    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118664    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118667    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118669    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118672    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118674    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118677    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118680    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118682    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118687    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118689    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118692    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118694    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118697    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118700    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118703    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118705    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:08.122588 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118708    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118712    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118715    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118718    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118721    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118724    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118727    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118729    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118732    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118735    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118738    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118740    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118743    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118745    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118748    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118750    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118753    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118755    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118758    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:08.123053 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118760    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118763    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118765    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118768    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118770    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118774    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118777    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118779    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118782    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118784    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118789    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118792    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118795    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118797    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118800    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118804    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118807    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118810    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118813    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:08.123528 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118815    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118818    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118820    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118824    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118826    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118829    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118831    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118834    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118836    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118838    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118841    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118844    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118846    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118848    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118851    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118854    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118856    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118858    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118862    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118865    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:08.123997 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118868    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118870    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.118873    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.118944    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.118984    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.118992    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.118996    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119000    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119004    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119008    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119013    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119016    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119019    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119023    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119027    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119030    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119033    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119037    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119039    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119042    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119046    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119050    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119053    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119056    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 16:16:08.124507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119059    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119062    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119065    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119068    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119074    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119078    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119082    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119084    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119087    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119091    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119094    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119098    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119101    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119104    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119106    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119110    2562 flags.go:64] FLAG: --enable-server="true"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119113    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119118    2562 flags.go:64] FLAG: --event-burst="100"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119121    2562 flags.go:64] FLAG: --event-qps="50"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119124    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119127    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119130    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119134    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119136    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119140    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 16:16:08.125084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119143    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119146    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119149    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119154    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119157    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119160    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119162    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119165    2562 flags.go:64] FLAG: --feature-gates=""
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119169    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119172    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119175    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119179    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119183    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119186    2562 flags.go:64] FLAG: --help="false"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119189    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119192    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119195    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119198    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119202    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119205    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119208    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119211    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119213    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 16:16:08.125704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119216    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119219    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119222    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119225    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119228    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119231    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119246    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119250    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119253    2562 flags.go:64] FLAG: --lock-file=""
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119256    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119260    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119263    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119271    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119274    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119277    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119280    2562 flags.go:64] FLAG: --logging-format="text"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119283    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119286    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119289    2562 flags.go:64] FLAG: --manifest-url=""
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119291    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119297    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119301    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119305    2562 flags.go:64] FLAG: --max-pods="110"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119308    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119312    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 16:16:08.126396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119314    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119317    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119321    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119324    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119327    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119334    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119337    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119341    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119344    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119346    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119352    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119355    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119359    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119361    2562 flags.go:64] FLAG: --port="10250"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119364    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119367    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04e47a97a2a04814a"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119371    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119374    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119378    2562 flags.go:64] FLAG: --register-node="true"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119382    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119386    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119389    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119392    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119395    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119398    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 16:16:08.127001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119402    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119405    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119408    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119412    2562 flags.go:64] FLAG: --runonce="false"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119415    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119418    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119421    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119424    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119426    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119430    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119433    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119436    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119439    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119442    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119445    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119448    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119451    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119454    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119457    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119462    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119465    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119468    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119472    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119475    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119477    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 16:16:08.127624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119480    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119485    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119488    2562 flags.go:64] FLAG: --v="2"
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119492    2562 flags.go:64] FLAG: --version="false"
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119496    2562 flags.go:64] FLAG: --vmodule=""
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119500    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.119503    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119624    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119628    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119632    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119638    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119642    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119646    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119649    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119654    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119657    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119660    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119663    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119666    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119669    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119672    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:08.128243 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119674    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119677    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119680    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119682    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119685    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119688    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119690    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119693    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119696    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119699    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119701    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119704    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119708    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119711    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119714    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119717    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119719    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119722    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119725    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119727    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:08.128765 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119730    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119734    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119737    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119740    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119743    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119746    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119748    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119751    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119754    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119756    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119759    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119762    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119764    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119767    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119770    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119773    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119775    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119778    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119781    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119783    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:08.129283 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119786    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119789    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119791    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119800    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119806    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119809    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119813    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119816    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119818    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119821    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119825    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119828    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119830    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119834    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119837    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119839    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119842    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119845    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119849    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119852    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:08.129810 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119856    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119859    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119862    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119864    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119867    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119870    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119872    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119875    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119877    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119880    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119882    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119885    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119887    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119890    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119893    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119895    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119899    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119902    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119904    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119908    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:08.130408 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119910    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119914    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119918    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119921    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.119924    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.120454    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.128343    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.128362    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.128384    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.128393    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128452    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128457    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128460    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128463    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:08.130888 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128467    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128469    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128472    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128475    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128477    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128481    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128483    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128486    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128489    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128492    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128494    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128497    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128499    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128502    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128505    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128508    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128510    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128513    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128516    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128518    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:08.131255 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128521    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128523    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128526    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128529    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128531    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128536    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128539    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128542    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128544    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128547    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128550    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128552    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128555    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128557    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128560    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128563    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128566    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128569    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128571    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128574    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:08.131759 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128577    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128579    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128582    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128585    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128587    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128590    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128592    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128596    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128598    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128601    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128603    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128606    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128609    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128612    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128629    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128633    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128636    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128639    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128643    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128646    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:08.132288 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128650    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128653    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128655    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128659    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128663    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128667    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128670    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128673    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128675    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128678    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128681    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128684    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128687    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128689    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128692    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128694    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128697    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128699    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128702    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:08.132766 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128704    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128708    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128712    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128715    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128718    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128722    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128726    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128729    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128732    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128736    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128739    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128742    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128745    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128748    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128751    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:08.133232 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128754    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.128759    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128879    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128884    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128887    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128890    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128893    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128896    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128899    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128902    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128905    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128907    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128911    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128915    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 16:16:08.133622 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128918    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128921    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128924    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128927    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128931    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128934    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128937    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128939    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128942    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128945    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128948    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128951    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128954    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128958    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128961    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128963    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128968    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128971    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128974    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128976    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 16:16:08.133970 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128979    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128981    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128984    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128986    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128989    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128992    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128994    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.128997    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129000    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129002    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129005    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129007    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129010    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129013    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129016    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129018    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129021    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129023    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129026    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 16:16:08.134534 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129029    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129031    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129034    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129037    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129040    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129042    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129045    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129048    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129051    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129053    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129057    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129060    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129062    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129065    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129068    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129070    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129073    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129075    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129078    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129081    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 16:16:08.134999 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129083    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129086    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129088    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129091    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129093    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129096    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129099    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129101    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129104    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129106    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129109    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129112    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129114    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129117    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129120    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129123    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129125    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129128    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129132    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129134    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 16:16:08.135619 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129137    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129140    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129144    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129148    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129151    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129154    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129157    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:08.129160    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.129165    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.129317    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.131378    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.132330    2562 server.go:1019] "Starting client certificate rotation"
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.132429    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:08.136463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.132473    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 16:16:08.136982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.134586    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:08.138588 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.138560    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 16:16:08.150219 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.150195    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 18 16:16:08.151820 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.151802    2562 log.go:25] "Validated CRI v1 image API"
Sep 18 16:16:08.153545 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.153526    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 16:16:08.156078 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.156055    2562 fs.go:135] Filesystem UUIDs: map[04b889ce-4e9b-42bf-8129-b1a2fe36e266:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fda32480-f684-4a70-9396-cd6178dc3ef4:/dev/nvme0n1p4]
Sep 18 16:16:08.156143 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.156075    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 16:16:08.160509 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.160385    2562 manager.go:216] Machine: {Timestamp:2026-09-18 16:16:08.159876132 +0000 UTC m=+1.935308300 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098817 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24f7f584227821f64b4186131bfda4 SystemUUID:ec24f7f5-8422-7821-f64b-4186131bfda4 BootID:37222339-bab3-4893-9b32-99d1cbf667a1 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:a1:0f:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:a1:0f:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:d1:d7:39:05:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 16:16:08.160509 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.160501    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 16:16:08.160644 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.160586    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 16:16:08.162665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.162635    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 16:16:08.162847 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.162667    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 16:16:08.163860 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.163846    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 16:16:08.163903 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.163864    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 16:16:08.163903 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.163872    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:08.163903 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.163883    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 16:16:08.165375 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.165360    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 16:16:08.169270 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.169249    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:08.169912 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.169898    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:08.170111 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170101    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 16:16:08.170146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170138    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 16:16:08.170173 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170162    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 16:16:08.170204 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170186    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 16:16:08.170204 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170201    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 18 16:16:08.170294 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170214    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 16:16:08.170692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170679    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:08.170748 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.170700    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 16:16:08.173141 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.173121    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 16:16:08.175477 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175463    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 16:16:08.175519 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175494    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 16:16:08.175671 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175661    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 16:16:08.175709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175676    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 16:16:08.175709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175682    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 16:16:08.175709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175688    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 16:16:08.175709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175695    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 16:16:08.175709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.175701    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 16:16:08.176139 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176124    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 16:16:08.176139 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176139    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 16:16:08.176299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176152    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 16:16:08.176299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176160    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 16:16:08.176299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176173    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 16:16:08.176299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176187    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 16:16:08.176299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176226    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 16:16:08.176299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.176247    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 16:16:08.180028 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.180010    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 16:16:08.184066 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.184046    2562 server.go:1268] "Started kubelet"
Sep 18 16:16:08.184204 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.184179    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 16:16:08.184279 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.184258    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 16:16:08.184684 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.184665    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 16:16:08.184937 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.184904    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 16:16:08.185023 ip-10-0-130-101 systemd[1]: Started Kubernetes Kubelet.
Sep 18 16:16:08.187440 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.187420    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 16:16:08.191320 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.191294    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 16:16:08.191320 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.191296    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 16:16:08.219815 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.219677    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 16:16:08.222267 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.194724    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:08.222987 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.222913    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.223076 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.223040    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 16:16:08.223374 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223350    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 16:16:08.223471 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223353    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 16:16:08.223471 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223427    2562 factory.go:55] Registering systemd factory
Sep 18 16:16:08.223471 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223443    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 18 16:16:08.223620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223433    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 16:16:08.223620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223574    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 16:16:08.223620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223582    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 16:16:08.223708 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223693    2562 factory.go:153] Registering CRI-O factory
Sep 18 16:16:08.223708 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223707    2562 factory.go:223] Registration of the crio container factory successfully
Sep 18 16:16:08.223759 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223752    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 16:16:08.223788 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223767    2562 factory.go:103] Registering Raw factory
Sep 18 16:16:08.223788 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.223779    2562 manager.go:1196] Started watching for new ooms in manager
Sep 18 16:16:08.224393 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.224380    2562 manager.go:349] Starting recovery of all containers
Sep 18 16:16:08.225840 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.225797    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 16:16:08.226195 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.226135    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 16:16:08.226623 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.225709    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-101.ec2.internal.18d67679fa03b8e0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-101.ec2.internal,UID:ip-10-0-130-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-101.ec2.internal,},FirstTimestamp:2026-09-18 16:16:08.184019168 +0000 UTC m=+1.959451320,LastTimestamp:2026-09-18 16:16:08.184019168 +0000 UTC m=+1.959451320,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-101.ec2.internal,}"
Sep 18 16:16:08.234684 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.234660    2562 manager.go:354] Recovery completed
Sep 18 16:16:08.236151 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.236130    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mpntt"
Sep 18 16:16:08.239442 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.239427    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:08.241336 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.241315    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:08.241462 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.241354    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:08.241462 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.241369    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:08.241987 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.241971    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 16:16:08.241987 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.241987    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 16:16:08.242142 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.241979    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:08.242142 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.242007    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 16:16:08.242743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.242727    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:08.242840 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.242759    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:08.242840 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.242774    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:08.244156 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.244134    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mpntt"
Sep 18 16:16:08.245095 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.245083    2562 policy_none.go:49] "None policy: Start"
Sep 18 16:16:08.245137 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.245100    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 16:16:08.245137 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.245111    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 16:16:08.247684 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.247668    2562 policy_none.go:47] "Start"
Sep 18 16:16:08.248312 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.248085    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-101.ec2.internal.18d67679fd6e4505  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-101.ec2.internal,UID:ip-10-0-130-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-101.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-101.ec2.internal,},FirstTimestamp:2026-09-18 16:16:08.241333509 +0000 UTC m=+2.016765662,LastTimestamp:2026-09-18 16:16:08.241333509 +0000 UTC m=+2.016765662,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-101.ec2.internal,}"
Sep 18 16:16:08.250253 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.250172    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-101.ec2.internal.18d67679fd6eafe7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-101.ec2.internal,UID:ip-10-0-130-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-101.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-101.ec2.internal,},FirstTimestamp:2026-09-18 16:16:08.241360871 +0000 UTC m=+2.016793027,LastTimestamp:2026-09-18 16:16:08.241360871 +0000 UTC m=+2.016793027,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-101.ec2.internal,}"
Sep 18 16:16:08.287042 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.287021    2562 manager.go:337] "Starting Device Plugin manager"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.287074    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.287088    2562 server.go:87] "Starting device plugin registration server"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.287395    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.287417    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.287534    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.287634    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.287642    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.288165    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 16:16:08.323567 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.288212    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.326423 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.326395    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 16:16:08.327652 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.327630    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 16:16:08.327765 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.327691    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 16:16:08.327765 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.327721    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 16:16:08.327868 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.327770    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 16:16:08.330230 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.330209    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:08.387812 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.387667    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:08.388910 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.388895    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:08.388991 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.388928    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:08.388991 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.388939    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:08.388991 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.388963    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.395904 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.395884    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.395964 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.395913    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-101.ec2.internal\": node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.428620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.428586    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal"]
Sep 18 16:16:08.428786 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.428664    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:08.430186 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.430165    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:08.430301 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.430198    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:08.430301 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.430211    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:08.431960 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.431936    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.432965 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.432952    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:08.433074 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.433133 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433093    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:08.433746 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433730    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:08.433844 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433753    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:08.433844 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433763    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:08.433844 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433771    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:08.433844 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433795    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:08.433844 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.433809    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:08.435996 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.435983    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.436046 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.436008    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 16:16:08.436731 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.436718    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 16:16:08.436792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.436745    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 16:16:08.436792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.436755    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeHasSufficientPID"
Sep 18 16:16:08.459320 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.459293    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-101.ec2.internal\" not found" node="ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.464081 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.464062    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-101.ec2.internal\" not found" node="ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.524877 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.524847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4627661586c3b25fe944f47481cab351-config\") pod \"kube-apiserver-proxy-ip-10-0-130-101.ec2.internal\" (UID: \"4627661586c3b25fe944f47481cab351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.524996 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.524884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7a2e2a2918026811ff0b4ef1d4f164d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal\" (UID: \"c7a2e2a2918026811ff0b4ef1d4f164d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.524996 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.524901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7a2e2a2918026811ff0b4ef1d4f164d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal\" (UID: \"c7a2e2a2918026811ff0b4ef1d4f164d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.532184 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.532157    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.625734 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.625700    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7a2e2a2918026811ff0b4ef1d4f164d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal\" (UID: \"c7a2e2a2918026811ff0b4ef1d4f164d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.625807 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.625738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7a2e2a2918026811ff0b4ef1d4f164d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal\" (UID: \"c7a2e2a2918026811ff0b4ef1d4f164d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.625807 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.625763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4627661586c3b25fe944f47481cab351-config\") pod \"kube-apiserver-proxy-ip-10-0-130-101.ec2.internal\" (UID: \"4627661586c3b25fe944f47481cab351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.625875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.625806    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4627661586c3b25fe944f47481cab351-config\") pod \"kube-apiserver-proxy-ip-10-0-130-101.ec2.internal\" (UID: \"4627661586c3b25fe944f47481cab351\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.625875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.625811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c7a2e2a2918026811ff0b4ef1d4f164d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal\" (UID: \"c7a2e2a2918026811ff0b4ef1d4f164d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.625875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.625811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7a2e2a2918026811ff0b4ef1d4f164d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal\" (UID: \"c7a2e2a2918026811ff0b4ef1d4f164d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.632842 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.632812    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.733702 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.733644    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.762837 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.762806    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.767688 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:08.767450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:08.833809 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.833763    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:08.934340 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:08.934310    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:09.034905 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.034836    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-101.ec2.internal\" not found"
Sep 18 16:16:09.081323 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.081295    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:09.123499 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.123465    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:09.131936 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.131913    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 16:16:09.132078 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.132056    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:09.132120 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.132094    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 16:16:09.132161 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.132132    2562 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a7fcdc15a722d441d84d411110ab9c85-cc4f6123aed674b7.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.130.101:33890->184.192.68.255:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:09.132191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.132166    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal"
Sep 18 16:16:09.148978 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.148950    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 16:16:09.171179 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.171152    2562 apiserver.go:52] "Watching apiserver"
Sep 18 16:16:09.179051 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.179025    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 16:16:09.179451 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.179427    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ds28f","openshift-network-operator/iptables-alerter-99rk5","openshift-ovn-kubernetes/ovnkube-node-gn65t","kube-system/global-pull-secret-syncer-mfqv2","kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal","openshift-cluster-node-tuning-operator/tuned-mqxks","openshift-dns/node-resolver-pz2lq","openshift-image-registry/node-ca-csgxn","openshift-multus/multus-additional-cni-plugins-4wz42","openshift-multus/network-metrics-daemon-8g777","openshift-network-diagnostics/network-check-target-8nj9b","kube-system/konnectivity-agent-sbb97","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal"]
Sep 18 16:16:09.180891 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.180876    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.183516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.183490    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-npkl6\""
Sep 18 16:16:09.183666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.183636    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 16:16:09.183666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.183641    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 16:16:09.183829 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.183787    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:09.183934 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.183884    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:09.186349 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.186329    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.187935 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.187917    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.188030 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.187976    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:09.188417 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.188403    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 16:16:09.188900 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.188885    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 16:16:09.189283 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.189271    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.189917 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.189877    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 16:16:09.189917 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.189888    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 16:16:09.189917 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.189890    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 16:16:09.190139 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.189918    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bdccn\""
Sep 18 16:16:09.190139 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.189899    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 16:16:09.190822 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.190807    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.191400 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.191383    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 16:16:09.191502 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.191490    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 16:16:09.191896 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.191876    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hth4q\""
Sep 18 16:16:09.192068 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.191879    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 16:16:09.192296 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.192277    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.193572 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.193557    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:09.193729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.193602    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b6sfz\""
Sep 18 16:16:09.193865 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.193845    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:09.193966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.193918    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.195189 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.195161    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 16:16:09.195299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.195231    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 16:16:09.195362 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.195329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 16:16:09.195362 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.195345    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 16:16:09.195520 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.195481    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sz7kv\""
Sep 18 16:16:09.195622 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.195531    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.195622 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.195568    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 16:16:09.196100 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.196082    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 16:16:09.196207 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.196193    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 16:16:09.196554 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.196322    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 16:16:09.196554 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.196322    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgjg8\""
Sep 18 16:16:09.196944 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.196930    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:09.197010 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.196989    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:09.198471 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.198454    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.198916 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.198758    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:16:09.198916 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.198848    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 16:16:09.198916 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.198891    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zx62l\""
Sep 18 16:16:09.199127 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.199111    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 16:16:09.200204 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.200185    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.200811 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.200796    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 16:16:09.201044 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.201030    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-98cgp\""
Sep 18 16:16:09.202427 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.202413    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hf8js\""
Sep 18 16:16:09.202508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.202492    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 16:16:09.202561 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.202511    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 16:16:09.223354 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.223317    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 16:16:09.224936 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.224916    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 16:16:09.229337 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6crgc\" (UniqueName: \"kubernetes.io/projected/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-kube-api-access-6crgc\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.229484 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229342    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-system-cni-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.229484 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-conf-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.229484 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysctl-conf\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.229484 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-host\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.229484 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229448    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-systemd-units\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.229687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229487    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-var-lib-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.229687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229538    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.229687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229580    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-systemd\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.229687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229602    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-kubelet\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.229687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-host\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.229687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229652    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysctl-d\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.229687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-var-lib-kubelet\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229696    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-node-log\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ddf424d-4391-4045-82cd-6e1bdc342467-kubelet-config\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229788    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1242b928-afef-4603-b4b1-f9d3ad271442-cni-binary-copy\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229802    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-k8s-cni-cncf-io\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-tuned\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229892    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-system-cni-dir\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-run-netns\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-cni-multus\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-lib-modules\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rwh\" (UniqueName: \"kubernetes.io/projected/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-kube-api-access-g5rwh\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.229992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1242b928-afef-4603-b4b1-f9d3ad271442-multus-daemon-config\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.230010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230011    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230035    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovnkube-script-lib\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230057    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-socket-dir-parent\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-modprobe-d\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230116    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysconfig\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230150    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-tmp\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ddf424d-4391-4045-82cd-6e1bdc342467-dbus\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-registration-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-sys\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-slash\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-log-socket\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-netns\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-hostroot\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstsq\" (UniqueName: \"kubernetes.io/projected/dbd37126-4128-4c29-a21f-3732d53d3548-kube-api-access-nstsq\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.230743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230411    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03205fd0-2fad-4887-9de5-1afa590637f2-agent-certs\") pod \"konnectivity-agent-sbb97\" (UID: \"03205fd0-2fad-4887-9de5-1afa590637f2\") " pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-serviceca\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhhx\" (UniqueName: \"kubernetes.io/projected/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-kube-api-access-kbhhx\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230537    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovn-node-metrics-cert\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-cni-bin\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03205fd0-2fad-4887-9de5-1afa590637f2-konnectivity-ca\") pod \"konnectivity-agent-sbb97\" (UID: \"03205fd0-2fad-4887-9de5-1afa590637f2\") " pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230607    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce80f8e8-94a6-4662-8531-c679fe377eef-hosts-file\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230632    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-cnibin\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-env-overrides\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-socket-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxp7r\" (UniqueName: \"kubernetes.io/projected/1242b928-afef-4603-b4b1-f9d3ad271442-kube-api-access-zxp7r\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce80f8e8-94a6-4662-8531-c679fe377eef-tmp-dir\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-etc-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230820    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-sys-fs\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.231508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230843    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-etc-kubernetes\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-kubernetes\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbd37126-4128-4c29-a21f-3732d53d3548-iptables-alerter-script\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-kubelet\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-cni-bin\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f6k\" (UniqueName: \"kubernetes.io/projected/df999121-c7c6-47d9-97f9-c4bc4b0fe062-kube-api-access-k4f6k\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.230980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-device-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-run\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpcrl\" (UniqueName: \"kubernetes.io/projected/ee46447e-bf78-431a-8433-2f1c5b570e78-kube-api-access-gpcrl\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbd37126-4128-4c29-a21f-3732d53d3548-host-slash\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-run-ovn-kubernetes\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-cni-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-cnibin\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-systemd\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-ovn\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.232157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231301    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.232701 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.232701 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtz79\" (UniqueName: \"kubernetes.io/projected/c5d52384-daab-49e1-8414-893a899382b1-kube-api-access-rtz79\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:09.232701 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231356    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-multus-certs\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.232701 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231377    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4j4v\" (UniqueName: \"kubernetes.io/projected/ce80f8e8-94a6-4662-8531-c679fe377eef-kube-api-access-v4j4v\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.232701 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231399    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-cni-netd\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.232701 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.231425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovnkube-config\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.235279 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.235260    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 16:16:09.250184 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.250120    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 16:11:08 +0000 UTC" deadline="2028-04-25 19:17:25.787436 +0000 UTC"
Sep 18 16:16:09.250184 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.250179    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14043h1m16.537259838s"
Sep 18 16:16:09.269185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.269156    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zcjwb"
Sep 18 16:16:09.275206 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.275184    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zcjwb"
Sep 18 16:16:09.332638 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332612    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-sys-fs\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.332638 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-etc-kubernetes\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-kubernetes\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332673    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbd37126-4128-4c29-a21f-3732d53d3548-iptables-alerter-script\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-kubelet\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-cni-bin\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332726    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-etc-kubernetes\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f6k\" (UniqueName: \"kubernetes.io/projected/df999121-c7c6-47d9-97f9-c4bc4b0fe062-kube-api-access-k4f6k\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-kubernetes\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-device-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-run\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332809    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-kubelet\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332820    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-sys-fs\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332831    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpcrl\" (UniqueName: \"kubernetes.io/projected/ee46447e-bf78-431a-8433-2f1c5b570e78-kube-api-access-gpcrl\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.332851 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332858    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbd37126-4128-4c29-a21f-3732d53d3548-host-slash\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332872    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-run\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-device-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-run-ovn-kubernetes\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-cni-bin\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332928    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-run-ovn-kubernetes\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332934    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-cni-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/dbd37126-4128-4c29-a21f-3732d53d3548-host-slash\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-cnibin\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.332984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-systemd\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333003    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-ovn\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333024    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-cnibin\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-systemd\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333072    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-ovn\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-cni-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.333549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtz79\" (UniqueName: \"kubernetes.io/projected/c5d52384-daab-49e1-8414-893a899382b1-kube-api-access-rtz79\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333107    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-multus-certs\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4j4v\" (UniqueName: \"kubernetes.io/projected/ce80f8e8-94a6-4662-8531-c679fe377eef-kube-api-access-v4j4v\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.333139    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333149    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-cni-netd\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333164    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-multus-certs\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovnkube-config\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6crgc\" (UniqueName: \"kubernetes.io/projected/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-kube-api-access-6crgc\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-system-cni-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-cni-netd\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-conf-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/dbd37126-4128-4c29-a21f-3732d53d3548-iptables-alerter-script\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333340    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-conf-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.333372    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret podName:0ddf424d-4391-4045-82cd-6e1bdc342467 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.833334051 +0000 UTC m=+3.608766203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret") pod "global-pull-secret-syncer-mfqv2" (UID: "0ddf424d-4391-4045-82cd-6e1bdc342467") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333339    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysctl-conf\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-host\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.334187 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333455    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-system-cni-dir\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333455    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-systemd-units\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-systemd-units\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333497    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-var-lib-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333523    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-host\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333556    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-var-lib-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333547    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-systemd\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-systemd\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333597    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-kubelet\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333613    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-host\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysctl-conf\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysctl-d\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333622    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-kubelet\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333654    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-var-lib-kubelet\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-host\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.335010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-etc-selinux\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333670    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-var-lib-kubelet\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-node-log\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovnkube-config\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysctl-d\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ddf424d-4391-4045-82cd-6e1bdc342467-kubelet-config\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333746    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0ddf424d-4391-4045-82cd-6e1bdc342467-kubelet-config\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333771    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1242b928-afef-4603-b4b1-f9d3ad271442-cni-binary-copy\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.333796    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333798    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-k8s-cni-cncf-io\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-tuned\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-node-log\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.333851    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.833835927 +0000 UTC m=+3.609268082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-system-cni-dir\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333894    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-run-netns\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.335818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333917    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-cni-multus\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-lib-modules\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rwh\" (UniqueName: \"kubernetes.io/projected/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-kube-api-access-g5rwh\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.333987    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1242b928-afef-4603-b4b1-f9d3ad271442-multus-daemon-config\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334038    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovnkube-script-lib\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-socket-dir-parent\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-modprobe-d\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334101    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysconfig\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334135    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-tmp\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ddf424d-4391-4045-82cd-6e1bdc342467-dbus\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-registration-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-sys\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-slash\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334286    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-log-socket\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-netns\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-hostroot\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334375    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-hostroot\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-system-cni-dir\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334419    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1242b928-afef-4603-b4b1-f9d3ad271442-cni-binary-copy\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0ddf424d-4391-4045-82cd-6e1bdc342467-dbus\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-sys\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-multus-socket-dir-parent\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334526    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-log-socket\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334101    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-k8s-cni-cncf-io\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334547    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-sysconfig\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334555    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-modprobe-d\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334570    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-cni-multus\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-registration-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334600    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-run-netns\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-slash\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.336838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334637    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstsq\" (UniqueName: \"kubernetes.io/projected/dbd37126-4128-4c29-a21f-3732d53d3548-kube-api-access-nstsq\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03205fd0-2fad-4887-9de5-1afa590637f2-agent-certs\") pod \"konnectivity-agent-sbb97\" (UID: \"03205fd0-2fad-4887-9de5-1afa590637f2\") " pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-lib-modules\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334736    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-serviceca\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334769    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-run-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-run-netns\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhhx\" (UniqueName: \"kubernetes.io/projected/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-kube-api-access-kbhhx\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334867    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334887    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovn-node-metrics-cert\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovnkube-script-lib\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-cni-bin\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03205fd0-2fad-4887-9de5-1afa590637f2-konnectivity-ca\") pod \"konnectivity-agent-sbb97\" (UID: \"03205fd0-2fad-4887-9de5-1afa590637f2\") " pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.334993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce80f8e8-94a6-4662-8531-c679fe377eef-hosts-file\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-cnibin\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335044    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-env-overrides\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.337490 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335069    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-socket-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335095    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxp7r\" (UniqueName: \"kubernetes.io/projected/1242b928-afef-4603-b4b1-f9d3ad271442-kube-api-access-zxp7r\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335121    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce80f8e8-94a6-4662-8531-c679fe377eef-tmp-dir\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335125    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee46447e-bf78-431a-8433-2f1c5b570e78-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335148    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-etc-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335158    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-serviceca\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335183    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1242b928-afef-4603-b4b1-f9d3ad271442-multus-daemon-config\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-cnibin\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce80f8e8-94a6-4662-8531-c679fe377eef-hosts-file\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335323    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/df999121-c7c6-47d9-97f9-c4bc4b0fe062-etc-openvswitch\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335351    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee46447e-bf78-431a-8433-2f1c5b570e78-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1242b928-afef-4603-b4b1-f9d3ad271442-host-var-lib-cni-bin\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335486    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-socket-dir\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/df999121-c7c6-47d9-97f9-c4bc4b0fe062-env-overrides\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335761    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce80f8e8-94a6-4662-8531-c679fe377eef-tmp-dir\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.335849    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/03205fd0-2fad-4887-9de5-1afa590637f2-konnectivity-ca\") pod \"konnectivity-agent-sbb97\" (UID: \"03205fd0-2fad-4887-9de5-1afa590637f2\") " pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.337567    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-etc-tuned\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.337586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-tmp\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.338034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.337643    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/df999121-c7c6-47d9-97f9-c4bc4b0fe062-ovn-node-metrics-cert\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.338656 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.337684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/03205fd0-2fad-4887-9de5-1afa590637f2-agent-certs\") pod \"konnectivity-agent-sbb97\" (UID: \"03205fd0-2fad-4887-9de5-1afa590637f2\") " pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.341883 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.341862    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:09.341883 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.341881    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:09.341883 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.341891    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dz85r for pod openshift-network-diagnostics/network-check-target-8nj9b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:09.342100 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.341944    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r podName:5629b342-03d2-4e24-84d4-14be4407f77b nodeName:}" failed. No retries permitted until 2026-09-18 16:16:09.8419316 +0000 UTC m=+3.617363745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dz85r" (UniqueName: "kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r") pod "network-check-target-8nj9b" (UID: "5629b342-03d2-4e24-84d4-14be4407f77b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:09.344700 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.344273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxp7r\" (UniqueName: \"kubernetes.io/projected/1242b928-afef-4603-b4b1-f9d3ad271442-kube-api-access-zxp7r\") pod \"multus-ds28f\" (UID: \"1242b928-afef-4603-b4b1-f9d3ad271442\") " pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.344700 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.344500    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpcrl\" (UniqueName: \"kubernetes.io/projected/ee46447e-bf78-431a-8433-2f1c5b570e78-kube-api-access-gpcrl\") pod \"multus-additional-cni-plugins-4wz42\" (UID: \"ee46447e-bf78-431a-8433-2f1c5b570e78\") " pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.344700 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.344655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtz79\" (UniqueName: \"kubernetes.io/projected/c5d52384-daab-49e1-8414-893a899382b1-kube-api-access-rtz79\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:09.344933 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.344903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4j4v\" (UniqueName: \"kubernetes.io/projected/ce80f8e8-94a6-4662-8531-c679fe377eef-kube-api-access-v4j4v\") pod \"node-resolver-pz2lq\" (UID: \"ce80f8e8-94a6-4662-8531-c679fe377eef\") " pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.345000 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.344983    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rwh\" (UniqueName: \"kubernetes.io/projected/ebcd7184-f371-4b4b-a3a4-263b26c0bbed-kube-api-access-g5rwh\") pod \"tuned-mqxks\" (UID: \"ebcd7184-f371-4b4b-a3a4-263b26c0bbed\") " pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.345281 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.345263    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f6k\" (UniqueName: \"kubernetes.io/projected/df999121-c7c6-47d9-97f9-c4bc4b0fe062-kube-api-access-k4f6k\") pod \"ovnkube-node-gn65t\" (UID: \"df999121-c7c6-47d9-97f9-c4bc4b0fe062\") " pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.345360 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.345297    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstsq\" (UniqueName: \"kubernetes.io/projected/dbd37126-4128-4c29-a21f-3732d53d3548-kube-api-access-nstsq\") pod \"iptables-alerter-99rk5\" (UID: \"dbd37126-4128-4c29-a21f-3732d53d3548\") " pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.346266 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.346227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6crgc\" (UniqueName: \"kubernetes.io/projected/e524fda2-859d-4b15-b1d9-d6c3d6454f0b-kube-api-access-6crgc\") pod \"aws-ebs-csi-driver-node-pzndw\" (UID: \"e524fda2-859d-4b15-b1d9-d6c3d6454f0b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.346584 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.346568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhhx\" (UniqueName: \"kubernetes.io/projected/5e1d1ef2-8d2a-447b-a817-d4c67229a9db-kube-api-access-kbhhx\") pod \"node-ca-csgxn\" (UID: \"5e1d1ef2-8d2a-447b-a817-d4c67229a9db\") " pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.350783 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.350753    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a2e2a2918026811ff0b4ef1d4f164d.slice/crio-a5e8c3bc250030c2dfc1441810251ac3770236c2d0e06c6c9a5dd1108a6b622f WatchSource:0}: Error finding container a5e8c3bc250030c2dfc1441810251ac3770236c2d0e06c6c9a5dd1108a6b622f: Status 404 returned error can't find the container with id a5e8c3bc250030c2dfc1441810251ac3770236c2d0e06c6c9a5dd1108a6b622f
Sep 18 16:16:09.351314 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.351295    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4627661586c3b25fe944f47481cab351.slice/crio-4f99545af1dfc4cb5139991d331a9fb9e7e4de61dd17421595af187039c9ced2 WatchSource:0}: Error finding container 4f99545af1dfc4cb5139991d331a9fb9e7e4de61dd17421595af187039c9ced2: Status 404 returned error can't find the container with id 4f99545af1dfc4cb5139991d331a9fb9e7e4de61dd17421595af187039c9ced2
Sep 18 16:16:09.355990 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.355976    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:16:09.426862 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.426831    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:09.508578 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.508544    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pz2lq"
Sep 18 16:16:09.517909 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.517882    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce80f8e8_94a6_4662_8531_c679fe377eef.slice/crio-1f1c5f9580787d728f04040cd32cbc8f162738f6b19621bce6819865afdde22f WatchSource:0}: Error finding container 1f1c5f9580787d728f04040cd32cbc8f162738f6b19621bce6819865afdde22f: Status 404 returned error can't find the container with id 1f1c5f9580787d728f04040cd32cbc8f162738f6b19621bce6819865afdde22f
Sep 18 16:16:09.533432 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.533407    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:09.538602 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.538578    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-csgxn"
Sep 18 16:16:09.541586 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.541559    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf999121_c7c6_47d9_97f9_c4bc4b0fe062.slice/crio-bd01817f38f7ed3e3fcc714553a7cb6507c1f64121356cdb5f2f223c725e0926 WatchSource:0}: Error finding container bd01817f38f7ed3e3fcc714553a7cb6507c1f64121356cdb5f2f223c725e0926: Status 404 returned error can't find the container with id bd01817f38f7ed3e3fcc714553a7cb6507c1f64121356cdb5f2f223c725e0926
Sep 18 16:16:09.546420 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.546397    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1d1ef2_8d2a_447b_a817_d4c67229a9db.slice/crio-2dc0d7fe9646c9cd70a7a939265196009b274f87fb4303dc6212cb7e54a1ae6c WatchSource:0}: Error finding container 2dc0d7fe9646c9cd70a7a939265196009b274f87fb4303dc6212cb7e54a1ae6c: Status 404 returned error can't find the container with id 2dc0d7fe9646c9cd70a7a939265196009b274f87fb4303dc6212cb7e54a1ae6c
Sep 18 16:16:09.553721 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.553698    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mqxks"
Sep 18 16:16:09.574416 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.574387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wz42"
Sep 18 16:16:09.580643 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.580618    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw"
Sep 18 16:16:09.582342 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.582314    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee46447e_bf78_431a_8433_2f1c5b570e78.slice/crio-3215af3bb60e3e502e176549f7916ebbd1bbf87c84c370b47f5a788fcc9b23f4 WatchSource:0}: Error finding container 3215af3bb60e3e502e176549f7916ebbd1bbf87c84c370b47f5a788fcc9b23f4: Status 404 returned error can't find the container with id 3215af3bb60e3e502e176549f7916ebbd1bbf87c84c370b47f5a788fcc9b23f4
Sep 18 16:16:09.588273 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.588231    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode524fda2_859d_4b15_b1d9_d6c3d6454f0b.slice/crio-748427f0f589f367081471be9ad7e457311a95cc0ebe5bfb729986dede021ae0 WatchSource:0}: Error finding container 748427f0f589f367081471be9ad7e457311a95cc0ebe5bfb729986dede021ae0: Status 404 returned error can't find the container with id 748427f0f589f367081471be9ad7e457311a95cc0ebe5bfb729986dede021ae0
Sep 18 16:16:09.605678 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.605645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-99rk5"
Sep 18 16:16:09.613685 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.613655    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd37126_4128_4c29_a21f_3732d53d3548.slice/crio-50e7a542e6ab6af20b8295eb30d3e1e424c666a20fdb2e0adc0d2aaf6ed487b4 WatchSource:0}: Error finding container 50e7a542e6ab6af20b8295eb30d3e1e424c666a20fdb2e0adc0d2aaf6ed487b4: Status 404 returned error can't find the container with id 50e7a542e6ab6af20b8295eb30d3e1e424c666a20fdb2e0adc0d2aaf6ed487b4
Sep 18 16:16:09.624475 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.624450    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ds28f"
Sep 18 16:16:09.630923 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.630899    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:09.632789 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.632766    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1242b928_afef_4603_b4b1_f9d3ad271442.slice/crio-d03e473c1f1b020a842b7534ca1aa3e9bbd0f06a57a6a8450702242a8d7fa6b4 WatchSource:0}: Error finding container d03e473c1f1b020a842b7534ca1aa3e9bbd0f06a57a6a8450702242a8d7fa6b4: Status 404 returned error can't find the container with id d03e473c1f1b020a842b7534ca1aa3e9bbd0f06a57a6a8450702242a8d7fa6b4
Sep 18 16:16:09.638370 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:09.638346    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03205fd0_2fad_4887_9de5_1afa590637f2.slice/crio-483b89d1630e60ab71fc2dd59665fd79a1709a79b0fcd2c479e2eb9414c82ab3 WatchSource:0}: Error finding container 483b89d1630e60ab71fc2dd59665fd79a1709a79b0fcd2c479e2eb9414c82ab3: Status 404 returned error can't find the container with id 483b89d1630e60ab71fc2dd59665fd79a1709a79b0fcd2c479e2eb9414c82ab3
Sep 18 16:16:09.748749 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.748717    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:09.839387 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.839297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:09.839552 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.839396    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:09.839552 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.839427    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:09.839552 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.839494    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:09.839552 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.839506    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.839486224 +0000 UTC m=+4.614918383 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:09.839552 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.839543    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret podName:0ddf424d-4391-4045-82cd-6e1bdc342467 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.839531597 +0000 UTC m=+4.614963741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret") pod "global-pull-secret-syncer-mfqv2" (UID: "0ddf424d-4391-4045-82cd-6e1bdc342467") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:09.940855 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:09.940816    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:09.941131 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.941112    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:09.941226 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.941141    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:09.941226 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.941154    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dz85r for pod openshift-network-diagnostics/network-check-target-8nj9b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:09.941226 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:09.941220    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r podName:5629b342-03d2-4e24-84d4-14be4407f77b nodeName:}" failed. No retries permitted until 2026-09-18 16:16:10.941194989 +0000 UTC m=+4.716627133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz85r" (UniqueName: "kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r") pod "network-check-target-8nj9b" (UID: "5629b342-03d2-4e24-84d4-14be4407f77b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:10.026613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.026575    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:10.277503 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.277390    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:09 +0000 UTC" deadline="2028-05-06 05:47:11.185485594 +0000 UTC"
Sep 18 16:16:10.277503 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.277453    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14293h31m0.908036968s"
Sep 18 16:16:10.329495 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.329463    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:10.329658 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.329603    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:10.361647 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.361479    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbb97" event={"ID":"03205fd0-2fad-4887-9de5-1afa590637f2","Type":"ContainerStarted","Data":"483b89d1630e60ab71fc2dd59665fd79a1709a79b0fcd2c479e2eb9414c82ab3"}
Sep 18 16:16:10.375128 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.374949    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99rk5" event={"ID":"dbd37126-4128-4c29-a21f-3732d53d3548","Type":"ContainerStarted","Data":"50e7a542e6ab6af20b8295eb30d3e1e424c666a20fdb2e0adc0d2aaf6ed487b4"}
Sep 18 16:16:10.394228 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.394034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqxks" event={"ID":"ebcd7184-f371-4b4b-a3a4-263b26c0bbed","Type":"ContainerStarted","Data":"b15da736c269bf53dba8af60ac9d8d6b43e258f857e1b6b8ee80a7311aacb44d"}
Sep 18 16:16:10.416948 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.416829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-csgxn" event={"ID":"5e1d1ef2-8d2a-447b-a817-d4c67229a9db","Type":"ContainerStarted","Data":"2dc0d7fe9646c9cd70a7a939265196009b274f87fb4303dc6212cb7e54a1ae6c"}
Sep 18 16:16:10.450564 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.450410    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"bd01817f38f7ed3e3fcc714553a7cb6507c1f64121356cdb5f2f223c725e0926"}
Sep 18 16:16:10.479516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.479474    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pz2lq" event={"ID":"ce80f8e8-94a6-4662-8531-c679fe377eef","Type":"ContainerStarted","Data":"1f1c5f9580787d728f04040cd32cbc8f162738f6b19621bce6819865afdde22f"}
Sep 18 16:16:10.497516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.497469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal" event={"ID":"4627661586c3b25fe944f47481cab351","Type":"ContainerStarted","Data":"4f99545af1dfc4cb5139991d331a9fb9e7e4de61dd17421595af187039c9ced2"}
Sep 18 16:16:10.514731 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.514690    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ds28f" event={"ID":"1242b928-afef-4603-b4b1-f9d3ad271442","Type":"ContainerStarted","Data":"d03e473c1f1b020a842b7534ca1aa3e9bbd0f06a57a6a8450702242a8d7fa6b4"}
Sep 18 16:16:10.525991 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.525955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw" event={"ID":"e524fda2-859d-4b15-b1d9-d6c3d6454f0b","Type":"ContainerStarted","Data":"748427f0f589f367081471be9ad7e457311a95cc0ebe5bfb729986dede021ae0"}
Sep 18 16:16:10.535817 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.535722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerStarted","Data":"3215af3bb60e3e502e176549f7916ebbd1bbf87c84c370b47f5a788fcc9b23f4"}
Sep 18 16:16:10.551736 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.551697    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal" event={"ID":"c7a2e2a2918026811ff0b4ef1d4f164d","Type":"ContainerStarted","Data":"a5e8c3bc250030c2dfc1441810251ac3770236c2d0e06c6c9a5dd1108a6b622f"}
Sep 18 16:16:10.851091 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.851000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:10.851091 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.851058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:10.851353 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.851177    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:10.851353 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.851252    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.85122021 +0000 UTC m=+6.626652367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:10.851658 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.851641    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:10.851815 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.851697    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret podName:0ddf424d-4391-4045-82cd-6e1bdc342467 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.851679001 +0000 UTC m=+6.627111158 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret") pod "global-pull-secret-syncer-mfqv2" (UID: "0ddf424d-4391-4045-82cd-6e1bdc342467") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:10.952133 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:10.952094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:10.952366 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.952284    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:10.952366 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.952319    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:10.952366 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.952333    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dz85r for pod openshift-network-diagnostics/network-check-target-8nj9b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:10.952546 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:10.952395    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r podName:5629b342-03d2-4e24-84d4-14be4407f77b nodeName:}" failed. No retries permitted until 2026-09-18 16:16:12.952377239 +0000 UTC m=+6.727809393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz85r" (UniqueName: "kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r") pod "network-check-target-8nj9b" (UID: "5629b342-03d2-4e24-84d4-14be4407f77b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:11.278648 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:11.278525    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 16:11:09 +0000 UTC" deadline="2028-03-09 07:29:09.386206132 +0000 UTC"
Sep 18 16:16:11.278648 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:11.278569    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12903h12m58.10764196s"
Sep 18 16:16:11.328117 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:11.328077    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:11.328330 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:11.328206    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:11.328642 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:11.328613    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:11.328794 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:11.328721    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:11.789417 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:11.789383    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 16:16:12.330620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:12.330589    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:12.331085 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.330728    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:12.871990 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:12.871945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:12.872184 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:12.872085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:12.872264 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.872213    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:12.872327 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.872294    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret podName:0ddf424d-4391-4045-82cd-6e1bdc342467 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.872274601 +0000 UTC m=+10.647706746 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret") pod "global-pull-secret-syncer-mfqv2" (UID: "0ddf424d-4391-4045-82cd-6e1bdc342467") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:12.872800 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.872782    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:12.872876 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.872840    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.872822894 +0000 UTC m=+10.648255054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:12.974060 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:12.973372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:12.974060 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.973535    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:12.974060 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.973559    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:12.974060 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.973571    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dz85r for pod openshift-network-diagnostics/network-check-target-8nj9b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:12.974060 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:12.973631    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r podName:5629b342-03d2-4e24-84d4-14be4407f77b nodeName:}" failed. No retries permitted until 2026-09-18 16:16:16.973612929 +0000 UTC m=+10.749045076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz85r" (UniqueName: "kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r") pod "network-check-target-8nj9b" (UID: "5629b342-03d2-4e24-84d4-14be4407f77b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:13.329620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:13.328856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:13.329620 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:13.328989    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:13.329620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:13.329410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:13.329620 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:13.329519    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:14.328365 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:14.328258    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:14.328836 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:14.328402    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:15.327944 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:15.327911    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:15.328132 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:15.328072    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:15.328417 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:15.327911    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:15.328417 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:15.328370    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:16.328652 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:16.328159    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:16.328652 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:16.328306    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:16.907146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:16.907114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:16.907360 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:16.907185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:16.907360 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:16.907295    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:16.907360 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:16.907300    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:16.907360 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:16.907359    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret podName:0ddf424d-4391-4045-82cd-6e1bdc342467 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.907333196 +0000 UTC m=+18.682765345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret") pod "global-pull-secret-syncer-mfqv2" (UID: "0ddf424d-4391-4045-82cd-6e1bdc342467") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:16.907580 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:16.907374    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:24.907367603 +0000 UTC m=+18.682799744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:17.008519 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:17.007895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:17.008519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:17.008060    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:17.008519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:17.008080    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:17.008519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:17.008092    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dz85r for pod openshift-network-diagnostics/network-check-target-8nj9b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:17.008519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:17.008148    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r podName:5629b342-03d2-4e24-84d4-14be4407f77b nodeName:}" failed. No retries permitted until 2026-09-18 16:16:25.00813116 +0000 UTC m=+18.783563316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz85r" (UniqueName: "kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r") pod "network-check-target-8nj9b" (UID: "5629b342-03d2-4e24-84d4-14be4407f77b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:17.329029 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:17.328481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:17.329029 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:17.328506    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:17.329029 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:17.328619    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:17.329029 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:17.328727    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:18.329955 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:18.329508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:18.329955 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:18.329600    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:19.328655 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:19.328608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:19.328822 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:19.328610    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:19.328822 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:19.328729    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:19.328895 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:19.328845    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:20.328656 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:20.328614    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:20.329124 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:20.328748    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:21.328656 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:21.328570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:21.328656 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:21.328613    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:21.329051 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:21.328716    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:21.329051 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:21.328846    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:22.328464 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:22.328430    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:22.328645 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:22.328553    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:23.328608 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:23.328569    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:23.329071 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:23.328573    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:23.329071 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:23.328729    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:23.329071 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:23.328813    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:24.328329 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:24.328288    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:24.328545 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:24.328400    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:24.965281 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:24.965209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:24.965788 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:24.965307    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:24.965788 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:24.965353    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:24.965788 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:24.965412    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:24.965788 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:24.965424    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.96540846 +0000 UTC m=+34.740840604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:24.965788 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:24.965467    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret podName:0ddf424d-4391-4045-82cd-6e1bdc342467 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:40.965451243 +0000 UTC m=+34.740883400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret") pod "global-pull-secret-syncer-mfqv2" (UID: "0ddf424d-4391-4045-82cd-6e1bdc342467") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:25.066054 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:25.066012    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:25.066267 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:25.066159    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:25.066267 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:25.066185    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:25.066267 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:25.066198    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dz85r for pod openshift-network-diagnostics/network-check-target-8nj9b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:25.066267 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:25.066267    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r podName:5629b342-03d2-4e24-84d4-14be4407f77b nodeName:}" failed. No retries permitted until 2026-09-18 16:16:41.066250111 +0000 UTC m=+34.841682272 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz85r" (UniqueName: "kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r") pod "network-check-target-8nj9b" (UID: "5629b342-03d2-4e24-84d4-14be4407f77b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:25.328132 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:25.328050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:25.328390 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:25.328173    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:25.328390 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:25.328224    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:25.328390 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:25.328344    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:26.328116 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:26.328067    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:26.328597 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:26.328209    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:27.329009 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:27.328969    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:27.329489 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:27.329121    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:27.329489 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:27.329169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:27.329489 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:27.329296    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:28.329123 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:28.329090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:28.329498 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:28.329177    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:29.328400 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.328088    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:29.328561 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.328100    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:29.328561 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:29.328497    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:29.328674 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:29.328559    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:29.600745 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.600659    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbb97" event={"ID":"03205fd0-2fad-4887-9de5-1afa590637f2","Type":"ContainerStarted","Data":"501cf932f4d2b9029930803528132d5614a8969d9030068130b51d1092b62a23"}
Sep 18 16:16:29.602216 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.602187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mqxks" event={"ID":"ebcd7184-f371-4b4b-a3a4-263b26c0bbed","Type":"ContainerStarted","Data":"101c43dcaea290e82f76fa33dd4c4a42261f36e029598719ebe53cdece99773e"}
Sep 18 16:16:29.603811 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.603772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-csgxn" event={"ID":"5e1d1ef2-8d2a-447b-a817-d4c67229a9db","Type":"ContainerStarted","Data":"a7966331250d3fb46d6f0c276f7a0384a5ae12741a8c7d3897f0edac80811787"}
Sep 18 16:16:29.607061 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.607035    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"7e4551c69762008cf1ab0a1484c2e1edb0c6ab80b4df5be44301ec40341af803"}
Sep 18 16:16:29.607181 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.607090    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"b83233ae6c6353c72b9c957ace8dc0ce973b1b034cb60dced2b85eccc7c10deb"}
Sep 18 16:16:29.607181 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.607121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"5bf7682911148c0598169c29a46481aa3c78877b3069f38d635ace7f1e67e288"}
Sep 18 16:16:29.607181 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.607151    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"8e05ba211560da96377d8f1533708916f54998804d55209696c5d32fff60428f"}
Sep 18 16:16:29.607322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.607185    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"c530f5813f825229ab3a8ee5ab6928b875081436bf62554211f0c39160a20bfe"}
Sep 18 16:16:29.607322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.607212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"7900cee90dd30f7606d754781a312a7fe3792be5f42214d2efe546c5f34778c0"}
Sep 18 16:16:29.608460 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.608436    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pz2lq" event={"ID":"ce80f8e8-94a6-4662-8531-c679fe377eef","Type":"ContainerStarted","Data":"34f6c567c5c9de418ada6d59a89c3d7efc84a87691c081b3a605f834d400ecbb"}
Sep 18 16:16:29.609876 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.609854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal" event={"ID":"4627661586c3b25fe944f47481cab351","Type":"ContainerStarted","Data":"636367d76f23adeb868d19db74d3d3ad513915938c6ee5f3d09ac84d87b978d0"}
Sep 18 16:16:29.611267 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.611231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ds28f" event={"ID":"1242b928-afef-4603-b4b1-f9d3ad271442","Type":"ContainerStarted","Data":"cc175e1286202193b176762b24eacc9672156681e3a0c10ac148a7df77e6d325"}
Sep 18 16:16:29.612668 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.612649    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw" event={"ID":"e524fda2-859d-4b15-b1d9-d6c3d6454f0b","Type":"ContainerStarted","Data":"c2fc9056bd5200277e44318a5f1630fecdc9727a382542114312beb3b5a536ee"}
Sep 18 16:16:29.614102 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.614083    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee46447e-bf78-431a-8433-2f1c5b570e78" containerID="b220b839478df77f80a4a1888008dade01e4e89276cf92911a6d70981f2a16d2" exitCode=0
Sep 18 16:16:29.614191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.614129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerDied","Data":"b220b839478df77f80a4a1888008dade01e4e89276cf92911a6d70981f2a16d2"}
Sep 18 16:16:29.615661 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.615644    2562 generic.go:356] "Generic (PLEG): container finished" podID="c7a2e2a2918026811ff0b4ef1d4f164d" containerID="a2dad9bdbf902ec385ce41fec1c9d602654f9c623dce52717a244dacbb37d106" exitCode=0
Sep 18 16:16:29.615709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.615674    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal" event={"ID":"c7a2e2a2918026811ff0b4ef1d4f164d","Type":"ContainerDied","Data":"a2dad9bdbf902ec385ce41fec1c9d602654f9c623dce52717a244dacbb37d106"}
Sep 18 16:16:29.631045 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.630991    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sbb97" podStartSLOduration=6.903909715 podStartE2EDuration="21.63097062s" totalImagesPullingTime="14.727060905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:29.616426909 +0000 UTC m=+23.391859071" watchObservedRunningTime="2026-09-18 16:16:29.63097062 +0000 UTC m=+23.406402784"
Sep 18 16:16:29.647255 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.647196    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-csgxn" podStartSLOduration=2.7384469769999997 podStartE2EDuration="21.647179515s" totalImagesPullingTime="18.908732538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:29.630841696 +0000 UTC m=+23.406273858" watchObservedRunningTime="2026-09-18 16:16:29.647179515 +0000 UTC m=+23.422611735"
Sep 18 16:16:29.647751 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.647726    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mqxks" podStartSLOduration=2.675193793 podStartE2EDuration="21.647719764s" totalImagesPullingTime="18.972525971s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:29.647147387 +0000 UTC m=+23.422579560" watchObservedRunningTime="2026-09-18 16:16:29.647719764 +0000 UTC m=+23.423151922"
Sep 18 16:16:29.665979 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.665924    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-101.ec2.internal" podStartSLOduration=20.665900497 podStartE2EDuration="20.665900497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:29.665365543 +0000 UTC m=+23.440797705" watchObservedRunningTime="2026-09-18 16:16:29.665900497 +0000 UTC m=+23.441332658"
Sep 18 16:16:29.681523 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.681476    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pz2lq" podStartSLOduration=2.7444034630000003 podStartE2EDuration="21.681460719s" totalImagesPullingTime="18.937057256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:29.680914343 +0000 UTC m=+23.456346498" watchObservedRunningTime="2026-09-18 16:16:29.681460719 +0000 UTC m=+23.456892881"
Sep 18 16:16:29.782983 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:29.782922    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ds28f" podStartSLOduration=2.8465776270000003 podStartE2EDuration="21.782905607s" totalImagesPullingTime="18.93632798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:29.782701654 +0000 UTC m=+23.558133839" watchObservedRunningTime="2026-09-18 16:16:29.782905607 +0000 UTC m=+23.558337769"
Sep 18 16:16:30.329207 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:30.328994    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:30.329399 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:30.329314    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:30.451330 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:30.451297    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 16:16:30.620425 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:30.620314    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-99rk5" event={"ID":"dbd37126-4128-4c29-a21f-3732d53d3548","Type":"ContainerStarted","Data":"0d5badc8fbe126e8761b0c1ea2f2b5fa9d82a7c2d3276546793e67a67f7214e8"}
Sep 18 16:16:30.622441 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:30.622403    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw" event={"ID":"e524fda2-859d-4b15-b1d9-d6c3d6454f0b","Type":"ContainerStarted","Data":"d1840e279897b2a7ca3cb90bcc6546c110a0d86af559904b8dbdf5d1fac9066d"}
Sep 18 16:16:30.639935 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:30.639884    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-99rk5" podStartSLOduration=3.720507315 podStartE2EDuration="22.639866623s" totalImagesPullingTime="18.919359308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:30.639253694 +0000 UTC m=+24.414685883" watchObservedRunningTime="2026-09-18 16:16:30.639866623 +0000 UTC m=+24.415298784"
Sep 18 16:16:31.300139 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.299988    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T16:16:30.451321534Z","UUID":"57a2e632-46e5-4036-bf6a-dacc41dc041d","Handler":null,"Name":"","Endpoint":""}
Sep 18 16:16:31.301869 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.301842    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 16:16:31.301869 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.301885    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 16:16:31.328164 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.328134    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:31.328341 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:31.328285    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:31.328482 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.328457    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:31.328595 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:31.328567    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:31.628985 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.628947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw" event={"ID":"e524fda2-859d-4b15-b1d9-d6c3d6454f0b","Type":"ContainerStarted","Data":"2b3bf68c4a07f841363dcd52a41cfb254a329e53aeaccd48a1959140dcbb4035"}
Sep 18 16:16:31.633642 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.633608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal" event={"ID":"c7a2e2a2918026811ff0b4ef1d4f164d","Type":"ContainerStarted","Data":"8a2f71dc2267d9acae3cc17bc00202b6cbc00bc37663d722c3825b397882d9d4"}
Sep 18 16:16:31.649167 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.649103    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pzndw" podStartSLOduration=2.832723911 podStartE2EDuration="23.649027233s" totalImagesPullingTime="20.816303322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:31.647636892 +0000 UTC m=+25.423069075" watchObservedRunningTime="2026-09-18 16:16:31.649027233 +0000 UTC m=+25.424459392"
Sep 18 16:16:31.665677 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:31.665625    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-101.ec2.internal" podStartSLOduration=22.665608757 podStartE2EDuration="22.665608757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:31.665071515 +0000 UTC m=+25.440503696" watchObservedRunningTime="2026-09-18 16:16:31.665608757 +0000 UTC m=+25.441040915"
Sep 18 16:16:32.328649 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:32.328616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:32.328814 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:32.328732    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:32.640404 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:32.640365    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"db9035996bcafc812055379ba7be8cdcfba67315e4f93f495326633619ccfd85"}
Sep 18 16:16:33.164889 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:33.164851    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:33.165561 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:33.165540    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:33.232337 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:33.232297    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:33.232922 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:33.232888    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sbb97"
Sep 18 16:16:33.328887 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:33.328851    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:33.329063 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:33.328984    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:33.329063 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:33.329033    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:33.329171 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:33.329144    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:34.328059 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:34.327878    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:34.328834 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:34.328129    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:34.646579 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:34.646547    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee46447e-bf78-431a-8433-2f1c5b570e78" containerID="c78872b1788585c5f791b38caef1b01b144c5679cbb2963c3ffd43c6a7914eb2" exitCode=0
Sep 18 16:16:34.646846 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:34.646635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerDied","Data":"c78872b1788585c5f791b38caef1b01b144c5679cbb2963c3ffd43c6a7914eb2"}
Sep 18 16:16:34.650587 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:34.650562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" event={"ID":"df999121-c7c6-47d9-97f9-c4bc4b0fe062","Type":"ContainerStarted","Data":"b71e50ecdd15a53e75accc15e7328e2652da52669664b96c2605df3408c0c72b"}
Sep 18 16:16:34.696865 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:34.696806    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t" podStartSLOduration=7.38888275 podStartE2EDuration="26.696789306s" totalImagesPullingTime="19.307906556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:34.695267972 +0000 UTC m=+28.470700149" watchObservedRunningTime="2026-09-18 16:16:34.696789306 +0000 UTC m=+28.472221467"
Sep 18 16:16:35.328453 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:35.328421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:35.328894 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:35.328421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:35.328894 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:35.328523    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:35.328894 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:35.328650    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:35.653104 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:35.653076    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:35.653267 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:35.653168    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:35.653267 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:35.653219    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:35.668863 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:35.668836    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:35.669006 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:35.668979    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:16:36.089653 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.089363    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mfqv2"]
Sep 18 16:16:36.089821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.089657    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:36.089821 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:36.089765    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:36.090099 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.090067    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8nj9b"]
Sep 18 16:16:36.090210 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.090194    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:36.090530 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:36.090311    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:36.102687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.102656    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8g777"]
Sep 18 16:16:36.102835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.102771    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:36.102893 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:36.102874    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:36.657214 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.657179    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee46447e-bf78-431a-8433-2f1c5b570e78" containerID="c647ed627155cbc74408b0b172a3a4e688daaaaffc78e66505dd9a641c3f1198" exitCode=0
Sep 18 16:16:36.657752 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:36.657265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerDied","Data":"c647ed627155cbc74408b0b172a3a4e688daaaaffc78e66505dd9a641c3f1198"}
Sep 18 16:16:37.328299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:37.328262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:37.328504 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:37.328384    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:38.328777 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:38.328738    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:38.329338 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:38.328840    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:38.329338 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:38.328871    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:38.329338 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:38.328942    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:38.663825 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:38.663794    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee46447e-bf78-431a-8433-2f1c5b570e78" containerID="2809d1d91c4420e20f68d28a0424559b487403a37f2a40bb31195cbdccce78bd" exitCode=0
Sep 18 16:16:38.663977 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:38.663841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerDied","Data":"2809d1d91c4420e20f68d28a0424559b487403a37f2a40bb31195cbdccce78bd"}
Sep 18 16:16:39.328682 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:39.328636    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:39.328863 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:39.328753    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:40.328581 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:40.328546    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:40.328882 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:40.328694    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:40.328882 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:40.328742    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:40.328882 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:40.328846    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:40.978036 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:40.977989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:40.978205 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:40.978059    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:40.978205 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:40.978134    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:40.978205 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:40.978180    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:40.978387 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:40.978210    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret podName:0ddf424d-4391-4045-82cd-6e1bdc342467 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:12.978195149 +0000 UTC m=+66.753627288 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret") pod "global-pull-secret-syncer-mfqv2" (UID: "0ddf424d-4391-4045-82cd-6e1bdc342467") : object "kube-system"/"original-pull-secret" not registered
Sep 18 16:16:40.978387 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:40.978253    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:12.978218342 +0000 UTC m=+66.753650491 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 16:16:41.079326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:41.079217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:41.079484 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:41.079408    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 16:16:41.079484 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:41.079434    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 16:16:41.079484 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:41.079444    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dz85r for pod openshift-network-diagnostics/network-check-target-8nj9b: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:41.079607 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:41.079502    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r podName:5629b342-03d2-4e24-84d4-14be4407f77b nodeName:}" failed. No retries permitted until 2026-09-18 16:17:13.079482675 +0000 UTC m=+66.854914818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz85r" (UniqueName: "kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r") pod "network-check-target-8nj9b" (UID: "5629b342-03d2-4e24-84d4-14be4407f77b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 16:16:41.328854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:41.328822    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:41.329037 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:41.328946    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:16:42.328608 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.328566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:42.328608 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.328592    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:42.328845 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:42.328717    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nj9b" podUID="5629b342-03d2-4e24-84d4-14be4407f77b"
Sep 18 16:16:42.328845 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:42.328806    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mfqv2" podUID="0ddf424d-4391-4045-82cd-6e1bdc342467"
Sep 18 16:16:42.563271 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.563168    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-101.ec2.internal" event="NodeReady"
Sep 18 16:16:42.563694 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.563363    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 16:16:42.608411 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.608373    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-km6fd"]
Sep 18 16:16:42.611548 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.611514    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-clq9q"]
Sep 18 16:16:42.611703 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.611616    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.615838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.614211    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f68fb\""
Sep 18 16:16:42.615838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.614291    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 16:16:42.616093 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.616056    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 16:16:42.616975 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.616943    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:42.619384 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.619358    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6h8pt\""
Sep 18 16:16:42.619532 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.619502    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 16:16:42.619620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.619599    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 16:16:42.620216 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.620196    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 16:16:42.621953 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.621515    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-km6fd"]
Sep 18 16:16:42.624315 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.624037    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clq9q"]
Sep 18 16:16:42.691820 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.691783    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d64b9b-0a6d-404b-9f0a-317263c530f4-tmp-dir\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.692022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.691844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdk5d\" (UniqueName: \"kubernetes.io/projected/a7d64b9b-0a6d-404b-9f0a-317263c530f4-kube-api-access-kdk5d\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.692022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.691876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfbp\" (UniqueName: \"kubernetes.io/projected/4f877dfe-a591-4047-b728-942d3115b0ab-kube-api-access-vvfbp\") pod \"ingress-canary-clq9q\" (UID: \"4f877dfe-a591-4047-b728-942d3115b0ab\") " pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:42.692022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.691901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.692022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.691943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d64b9b-0a6d-404b-9f0a-317263c530f4-config-volume\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.692022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.691973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f877dfe-a591-4047-b728-942d3115b0ab-cert\") pod \"ingress-canary-clq9q\" (UID: \"4f877dfe-a591-4047-b728-942d3115b0ab\") " pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:42.792663 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.792621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f877dfe-a591-4047-b728-942d3115b0ab-cert\") pod \"ingress-canary-clq9q\" (UID: \"4f877dfe-a591-4047-b728-942d3115b0ab\") " pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:42.792843 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.792808    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d64b9b-0a6d-404b-9f0a-317263c530f4-tmp-dir\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.792917 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.792872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdk5d\" (UniqueName: \"kubernetes.io/projected/a7d64b9b-0a6d-404b-9f0a-317263c530f4-kube-api-access-kdk5d\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.792917 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.792909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfbp\" (UniqueName: \"kubernetes.io/projected/4f877dfe-a591-4047-b728-942d3115b0ab-kube-api-access-vvfbp\") pod \"ingress-canary-clq9q\" (UID: \"4f877dfe-a591-4047-b728-942d3115b0ab\") " pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:42.793020 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.792933    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.793020 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.792988    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d64b9b-0a6d-404b-9f0a-317263c530f4-config-volume\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.793268 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.793223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7d64b9b-0a6d-404b-9f0a-317263c530f4-tmp-dir\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.793367 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:42.793355    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:42.793423 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:42.793411    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:43.293393319 +0000 UTC m=+37.068825462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:42.793607 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.793586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a7d64b9b-0a6d-404b-9f0a-317263c530f4-config-volume\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.797793 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.797619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f877dfe-a591-4047-b728-942d3115b0ab-cert\") pod \"ingress-canary-clq9q\" (UID: \"4f877dfe-a591-4047-b728-942d3115b0ab\") " pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:42.804482 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.804453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdk5d\" (UniqueName: \"kubernetes.io/projected/a7d64b9b-0a6d-404b-9f0a-317263c530f4-kube-api-access-kdk5d\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:42.804632 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.804608    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfbp\" (UniqueName: \"kubernetes.io/projected/4f877dfe-a591-4047-b728-942d3115b0ab-kube-api-access-vvfbp\") pod \"ingress-canary-clq9q\" (UID: \"4f877dfe-a591-4047-b728-942d3115b0ab\") " pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:42.935108 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:42.934797    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-clq9q"
Sep 18 16:16:43.224226 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.224140    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-clq9q"]
Sep 18 16:16:43.227783 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:16:43.227738    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f877dfe_a591_4047_b728_942d3115b0ab.slice/crio-dfe58c53515abb3f5d822de4ae240b86582c5566fdc82469208c168900ddbfd3 WatchSource:0}: Error finding container dfe58c53515abb3f5d822de4ae240b86582c5566fdc82469208c168900ddbfd3: Status 404 returned error can't find the container with id dfe58c53515abb3f5d822de4ae240b86582c5566fdc82469208c168900ddbfd3
Sep 18 16:16:43.296458 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.296422    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:43.296610 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:43.296569    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:43.296650 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:43.296633    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:44.296613377 +0000 UTC m=+38.072045521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:43.328818 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.328786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:16:43.331727 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.331691    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:16:43.331852 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.331746    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-px2dk\""
Sep 18 16:16:43.678817 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.678780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clq9q" event={"ID":"4f877dfe-a591-4047-b728-942d3115b0ab","Type":"ContainerStarted","Data":"dfe58c53515abb3f5d822de4ae240b86582c5566fdc82469208c168900ddbfd3"}
Sep 18 16:16:43.682011 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.681974    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee46447e-bf78-431a-8433-2f1c5b570e78" containerID="568c20797aa2b4c1813b4797373c004224e5a3424ee2d6a01727e6aa1ee00b36" exitCode=0
Sep 18 16:16:43.682139 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:43.682027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerDied","Data":"568c20797aa2b4c1813b4797373c004224e5a3424ee2d6a01727e6aa1ee00b36"}
Sep 18 16:16:44.304828 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.304790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:44.305002 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:44.304943    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:44.305057 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:44.305016    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:46.304999869 +0000 UTC m=+40.080432009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:44.328643 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.328608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:16:44.328801 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.328608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:16:44.331279 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.331255    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:16:44.332218 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.332175    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:16:44.332341 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.332220    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:16:44.332341 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.332320    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fcv6z\""
Sep 18 16:16:44.689279 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.689229    2562 generic.go:356] "Generic (PLEG): container finished" podID="ee46447e-bf78-431a-8433-2f1c5b570e78" containerID="e09b4cceb830492c991dc2425cc54ade89b167b30113e771108b7abb464c0e36" exitCode=0
Sep 18 16:16:44.689279 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:44.689273    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerDied","Data":"e09b4cceb830492c991dc2425cc54ade89b167b30113e771108b7abb464c0e36"}
Sep 18 16:16:45.693958 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:45.693740    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-clq9q" event={"ID":"4f877dfe-a591-4047-b728-942d3115b0ab","Type":"ContainerStarted","Data":"c696485976165be0edfc1209051ab58da3a7cf0684bcceda6bb36e3ce9990613"}
Sep 18 16:16:45.699463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:45.699431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wz42" event={"ID":"ee46447e-bf78-431a-8433-2f1c5b570e78","Type":"ContainerStarted","Data":"5807682663a72c958c48b88672016d8eea01902f3e4539d69aba36df29de168d"}
Sep 18 16:16:45.710669 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:45.710618    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-clq9q" podStartSLOduration=1.477396551 podStartE2EDuration="3.710600954s" totalImagesPullingTime="2.233204403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:45.709561788 +0000 UTC m=+39.484993949" watchObservedRunningTime="2026-09-18 16:16:45.710600954 +0000 UTC m=+39.486033115"
Sep 18 16:16:45.745378 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:45.745320    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4wz42" podStartSLOduration=7.767213948 podStartE2EDuration="37.74529711s" totalImagesPullingTime="29.978083162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:16:45.743830372 +0000 UTC m=+39.519262553" watchObservedRunningTime="2026-09-18 16:16:45.74529711 +0000 UTC m=+39.520729273"
Sep 18 16:16:46.320951 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:46.320903    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:46.321125 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:46.321071    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:46.321173 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:46.321136    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:50.321121106 +0000 UTC m=+44.096553251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:50.348756 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:50.348717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:50.349263 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:50.348842    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:50.349263 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:50.348894    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:16:58.348882357 +0000 UTC m=+52.124314498 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:16:58.408158 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:16:58.408109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:16:58.408600 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:58.408290    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:16:58.408600 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:16:58.408362    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:14.408345113 +0000 UTC m=+68.183777258 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:07.670275 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:07.670228    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gn65t"
Sep 18 16:17:13.007386 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.007346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:17:13.007792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.007429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:17:13.011758 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.011740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 16:17:13.011843 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.011756    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 16:17:13.017840 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:17:13.017813    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:17:13.017932 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:17:13.017877    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:18:17.017860461 +0000 UTC m=+130.793292606 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : secret "metrics-daemon-secret" not found
Sep 18 16:17:13.021024 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.021001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0ddf424d-4391-4045-82cd-6e1bdc342467-original-pull-secret\") pod \"global-pull-secret-syncer-mfqv2\" (UID: \"0ddf424d-4391-4045-82cd-6e1bdc342467\") " pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:17:13.108310 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.108270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:17:13.111593 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.111571    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 16:17:13.122104 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.122073    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 16:17:13.131530 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.131506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz85r\" (UniqueName: \"kubernetes.io/projected/5629b342-03d2-4e24-84d4-14be4407f77b-kube-api-access-dz85r\") pod \"network-check-target-8nj9b\" (UID: \"5629b342-03d2-4e24-84d4-14be4407f77b\") " pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:17:13.141522 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.141491    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mfqv2"
Sep 18 16:17:13.150664 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.150633    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fcv6z\""
Sep 18 16:17:13.158398 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.158359    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:17:13.443217 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.443183    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mfqv2"]
Sep 18 16:17:13.446476 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:17:13.446452    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddf424d_4391_4045_82cd_6e1bdc342467.slice/crio-935a7ecf8e41c96b20eafa4722cda199babb99c258182c2f08c408c197d116f2 WatchSource:0}: Error finding container 935a7ecf8e41c96b20eafa4722cda199babb99c258182c2f08c408c197d116f2: Status 404 returned error can't find the container with id 935a7ecf8e41c96b20eafa4722cda199babb99c258182c2f08c408c197d116f2
Sep 18 16:17:13.462614 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.462588    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8nj9b"]
Sep 18 16:17:13.467045 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:17:13.467017    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5629b342_03d2_4e24_84d4_14be4407f77b.slice/crio-aa41ea56f809218f3ffb4eaa9aabdd6199962a3661d63688ccf216ad3ead9cea WatchSource:0}: Error finding container aa41ea56f809218f3ffb4eaa9aabdd6199962a3661d63688ccf216ad3ead9cea: Status 404 returned error can't find the container with id aa41ea56f809218f3ffb4eaa9aabdd6199962a3661d63688ccf216ad3ead9cea
Sep 18 16:17:13.766920 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.766823    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8nj9b" event={"ID":"5629b342-03d2-4e24-84d4-14be4407f77b","Type":"ContainerStarted","Data":"aa41ea56f809218f3ffb4eaa9aabdd6199962a3661d63688ccf216ad3ead9cea"}
Sep 18 16:17:13.768069 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:13.768041    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mfqv2" event={"ID":"0ddf424d-4391-4045-82cd-6e1bdc342467","Type":"ContainerStarted","Data":"935a7ecf8e41c96b20eafa4722cda199babb99c258182c2f08c408c197d116f2"}
Sep 18 16:17:14.418366 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:14.418327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:17:14.418843 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:17:14.418659    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:14.418843 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:17:14.418726    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:17:46.41870693 +0000 UTC m=+100.194139072 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:18.783599 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:18.783556    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8nj9b" event={"ID":"5629b342-03d2-4e24-84d4-14be4407f77b","Type":"ContainerStarted","Data":"bc1aca5d697ad1ad4a0a1cc954dcb1af9c4bab6a5421f1264d562b080bbe9c06"}
Sep 18 16:17:18.784102 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:18.783638    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:17:18.785089 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:18.785064    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mfqv2" event={"ID":"0ddf424d-4391-4045-82cd-6e1bdc342467","Type":"ContainerStarted","Data":"4c497f03682ce344c68d2f95ed768d82b50c061f5949eca4b9d48456043a448d"}
Sep 18 16:17:18.802729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:18.802677    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8nj9b" podStartSLOduration=66.432682293 podStartE2EDuration="1m10.80266291s" totalImagesPullingTime="4.369980617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:18.802317535 +0000 UTC m=+72.577749712" watchObservedRunningTime="2026-09-18 16:17:18.80266291 +0000 UTC m=+72.578095072"
Sep 18 16:17:18.820176 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:18.820121    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mfqv2" podStartSLOduration=66.42564871 podStartE2EDuration="1m10.82010596s" totalImagesPullingTime="4.39445725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:17:18.819393988 +0000 UTC m=+72.594826150" watchObservedRunningTime="2026-09-18 16:17:18.82010596 +0000 UTC m=+72.595538121"
Sep 18 16:17:46.438796 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:46.438746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:17:46.439222 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:17:46.438882    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:17:46.439222 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:17:46.438958    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:18:50.438941865 +0000 UTC m=+164.214374005 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:17:49.791050 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:17:49.791019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8nj9b"
Sep 18 16:18:17.059188 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:18:17.059124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:18:17.059624 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:18:17.059288    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 16:18:17.059624 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:18:17.059353    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs podName:c5d52384-daab-49e1-8414-893a899382b1 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:19.059338155 +0000 UTC m=+252.834770311 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs") pod "network-metrics-daemon-8g777" (UID: "c5d52384-daab-49e1-8414-893a899382b1") : secret "metrics-daemon-secret" not found
Sep 18 16:18:45.627927 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:18:45.627877    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-km6fd" podUID="a7d64b9b-0a6d-404b-9f0a-317263c530f4"
Sep 18 16:18:46.002533 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:18:46.002502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-km6fd"
Sep 18 16:18:46.337511 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:18:46.337425    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8g777" podUID="c5d52384-daab-49e1-8414-893a899382b1"
Sep 18 16:18:50.475772 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:18:50.475680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:18:50.476202 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:18:50.475835    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 16:18:50.476202 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:18:50.475904    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls podName:a7d64b9b-0a6d-404b-9f0a-317263c530f4 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:52.475888684 +0000 UTC m=+286.251320829 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls") pod "dns-default-km6fd" (UID: "a7d64b9b-0a6d-404b-9f0a-317263c530f4") : secret "dns-default-metrics-tls" not found
Sep 18 16:18:57.327967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:18:57.327922    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:19:53.567974 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.567939    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"]
Sep 18 16:19:53.570266 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.570231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"
Sep 18 16:19:53.573071 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.573045    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"]
Sep 18 16:19:53.573647 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.573623    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.573780 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.573642    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zstbs\""
Sep 18 16:19:53.573979 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.573964    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.575020 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.575000    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"
Sep 18 16:19:53.577371 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.577352    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nwp65\""
Sep 18 16:19:53.598339 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.598309    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"]
Sep 18 16:19:53.610083 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.610043    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"]
Sep 18 16:19:53.680902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.680870    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"]
Sep 18 16:19:53.682664 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.682646    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.688080 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.688049    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 18 16:19:53.689495 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.689475    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-656b55b86d-nx575"]
Sep 18 16:19:53.690947 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.690927    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 18 16:19:53.691212 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.691195    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.691279 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.691259    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.695436 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.695416    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-b6rj2\""
Sep 18 16:19:53.695668 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.695654    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.696168 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.696146    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 16:19:53.698501 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.698473    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 18 16:19:53.698639 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.698500    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 18 16:19:53.698814 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.698792    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 16:19:53.698906 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.698814    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 18 16:19:53.698906 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.698805    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 18 16:19:53.700048 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.700032    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fgfg6\""
Sep 18 16:19:53.702002 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.701980    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"]
Sep 18 16:19:53.706073 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.706047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xmx\" (UniqueName: \"kubernetes.io/projected/3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0-kube-api-access-j2xmx\") pod \"network-check-source-7cbc45fddc-729fd\" (UID: \"3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"
Sep 18 16:19:53.706196 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.706087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9c2\" (UniqueName: \"kubernetes.io/projected/f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc-kube-api-access-2m9c2\") pod \"volume-data-source-validator-5c4db8bdf9-88s9f\" (UID: \"f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"
Sep 18 16:19:53.715648 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.715609    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656b55b86d-nx575"]
Sep 18 16:19:53.806970 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.806930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gj8t\" (UniqueName: \"kubernetes.io/projected/bedebae7-23bb-4bb2-8550-b66910de6e6d-kube-api-access-4gj8t\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.806970 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.806974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9c2\" (UniqueName: \"kubernetes.io/projected/f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc-kube-api-access-2m9c2\") pod \"volume-data-source-validator-5c4db8bdf9-88s9f\" (UID: \"f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"
Sep 18 16:19:53.807193 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.807193 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-stats-auth\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.807193 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807054    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.807193 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7ll\" (UniqueName: \"kubernetes.io/projected/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-kube-api-access-7l7ll\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.807193 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807148    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.807193 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807186    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-default-certificate\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.807408 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807231    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.807408 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.807297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xmx\" (UniqueName: \"kubernetes.io/projected/3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0-kube-api-access-j2xmx\") pod \"network-check-source-7cbc45fddc-729fd\" (UID: \"3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"
Sep 18 16:19:53.815459 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.815430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xmx\" (UniqueName: \"kubernetes.io/projected/3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0-kube-api-access-j2xmx\") pod \"network-check-source-7cbc45fddc-729fd\" (UID: \"3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"
Sep 18 16:19:53.815459 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.815436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9c2\" (UniqueName: \"kubernetes.io/projected/f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc-kube-api-access-2m9c2\") pod \"volume-data-source-validator-5c4db8bdf9-88s9f\" (UID: \"f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"
Sep 18 16:19:53.879713 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.879680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"
Sep 18 16:19:53.884922 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.884891    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"
Sep 18 16:19:53.908474 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.908665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7ll\" (UniqueName: \"kubernetes.io/projected/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-kube-api-access-7l7ll\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.908665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.908665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-default-certificate\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.908665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.908665 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:53.908617    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:19:53.908909 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:53.908697    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:19:54.408674691 +0000 UTC m=+228.184106842 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : secret "router-metrics-certs-default" not found
Sep 18 16:19:53.908909 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gj8t\" (UniqueName: \"kubernetes.io/projected/bedebae7-23bb-4bb2-8550-b66910de6e6d-kube-api-access-4gj8t\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.908909 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.908909 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:53.908814    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:19:54.408794292 +0000 UTC m=+228.184226438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : configmap references non-existent config key: service-ca.crt
Sep 18 16:19:53.908909 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.908842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-stats-auth\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.909208 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.909184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-config\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.911560 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.911531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-serving-cert\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.911709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.911564    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-stats-auth\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.911984 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.911961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-default-certificate\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.918027 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.917958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gj8t\" (UniqueName: \"kubernetes.io/projected/bedebae7-23bb-4bb2-8550-b66910de6e6d-kube-api-access-4gj8t\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:53.918987 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.918960    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7ll\" (UniqueName: \"kubernetes.io/projected/362c0f6a-10ae-45e0-bcc4-ecddfbea6a64-kube-api-access-7l7ll\") pod \"kube-storage-version-migrator-operator-55889f7b47-l4lld\" (UID: \"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:53.991550 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:53.991508    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"
Sep 18 16:19:54.154004 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:54.153917    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f"]
Sep 18 16:19:54.157450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:54.157416    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd"]
Sep 18 16:19:54.157670 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:19:54.157642    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b3b2cf_1efa_4c5b_9a1f_499dc21290fc.slice/crio-a1bb3b24b12ed30de92b388b3ec887f09d960ead3e554ee597d4622c2be407c9 WatchSource:0}: Error finding container a1bb3b24b12ed30de92b388b3ec887f09d960ead3e554ee597d4622c2be407c9: Status 404 returned error can't find the container with id a1bb3b24b12ed30de92b388b3ec887f09d960ead3e554ee597d4622c2be407c9
Sep 18 16:19:54.161648 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:19:54.161620    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e346a5a_c4a0_40c4_9d99_6c97a5e6f1d0.slice/crio-b3be0a9910c6959cd6bef73df95e25fe3c52a72177414daa2e54509ae46c9e55 WatchSource:0}: Error finding container b3be0a9910c6959cd6bef73df95e25fe3c52a72177414daa2e54509ae46c9e55: Status 404 returned error can't find the container with id b3be0a9910c6959cd6bef73df95e25fe3c52a72177414daa2e54509ae46c9e55
Sep 18 16:19:54.171263 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:54.171215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd" event={"ID":"3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0","Type":"ContainerStarted","Data":"b3be0a9910c6959cd6bef73df95e25fe3c52a72177414daa2e54509ae46c9e55"}
Sep 18 16:19:54.172430 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:54.172395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f" event={"ID":"f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc","Type":"ContainerStarted","Data":"a1bb3b24b12ed30de92b388b3ec887f09d960ead3e554ee597d4622c2be407c9"}
Sep 18 16:19:54.260537 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:54.260499    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld"]
Sep 18 16:19:54.263959 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:19:54.263929    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362c0f6a_10ae_45e0_bcc4_ecddfbea6a64.slice/crio-9b8c6e58c6a417ac0fc23de4714d6b936e4433d2119f449c74876d0b4e59b136 WatchSource:0}: Error finding container 9b8c6e58c6a417ac0fc23de4714d6b936e4433d2119f449c74876d0b4e59b136: Status 404 returned error can't find the container with id 9b8c6e58c6a417ac0fc23de4714d6b936e4433d2119f449c74876d0b4e59b136
Sep 18 16:19:54.412265 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:54.412145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:54.412265 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:54.412259    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:54.412499 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:54.412346    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:19:55.412327588 +0000 UTC m=+229.187759729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : configmap references non-existent config key: service-ca.crt
Sep 18 16:19:54.412499 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:54.412386    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:19:54.412499 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:54.412443    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:19:55.41242799 +0000 UTC m=+229.187860133 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : secret "router-metrics-certs-default" not found
Sep 18 16:19:55.177160 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:55.177125    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld" event={"ID":"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64","Type":"ContainerStarted","Data":"9b8c6e58c6a417ac0fc23de4714d6b936e4433d2119f449c74876d0b4e59b136"}
Sep 18 16:19:55.178726 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:55.178700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd" event={"ID":"3e346a5a-c4a0-40c4-9d99-6c97a5e6f1d0","Type":"ContainerStarted","Data":"5c66d8f89d62da78b35c41946b4135ea8b7efe2e5aef39fddee3bd54f8bcc0fe"}
Sep 18 16:19:55.198929 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:55.197033    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-729fd" podStartSLOduration=2.197010893 podStartE2EDuration="2.197010893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:55.19418081 +0000 UTC m=+228.969612991" watchObservedRunningTime="2026-09-18 16:19:55.197010893 +0000 UTC m=+228.972443056"
Sep 18 16:19:55.418127 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:55.418086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:55.418314 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:55.418161    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:55.418314 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:55.418279    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:19:55.418486 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:55.418355    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:19:57.418336343 +0000 UTC m=+231.193768485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : secret "router-metrics-certs-default" not found
Sep 18 16:19:55.418486 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:55.418386    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:19:57.418361205 +0000 UTC m=+231.193793357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : configmap references non-existent config key: service-ca.crt
Sep 18 16:19:56.182897 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:56.182851    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f" event={"ID":"f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc","Type":"ContainerStarted","Data":"033263c0149e120ebda66f6a0dd19229d70edf1a2eb59a7a402d72375a25c9f2"}
Sep 18 16:19:56.198760 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:56.198697    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-88s9f" podStartSLOduration=1.9294297550000001 podStartE2EDuration="3.198672114s" totalImagesPullingTime="1.269242359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:56.197685298 +0000 UTC m=+229.973117481" watchObservedRunningTime="2026-09-18 16:19:56.198672114 +0000 UTC m=+229.974104278"
Sep 18 16:19:57.188030 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.187993    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld" event={"ID":"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64","Type":"ContainerStarted","Data":"20b84da5f214766eb81e8004fdeacec4f3b0c554ccf1751ea786e94b19fcfc51"}
Sep 18 16:19:57.204355 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.204299    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld" podStartSLOduration=1.8665258850000002 podStartE2EDuration="4.204279983s" totalImagesPullingTime="2.337754098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:19:57.203804626 +0000 UTC m=+230.979236789" watchObservedRunningTime="2026-09-18 16:19:57.204279983 +0000 UTC m=+230.979712191"
Sep 18 16:19:57.430937 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.430890    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:57.431121 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.430957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:19:57.431121 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:57.431027    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:19:57.431121 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:57.431094    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.431078652 +0000 UTC m=+235.206510798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : secret "router-metrics-certs-default" not found
Sep 18 16:19:57.431121 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:57.431111    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.431103543 +0000 UTC m=+235.206535682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : configmap references non-existent config key: service-ca.crt
Sep 18 16:19:57.643841 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.643801    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8prt9"]
Sep 18 16:19:57.646144 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.646121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.649199 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.649176    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 16:19:57.649325 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.649178    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 16:19:57.649960 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.649784    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 16:19:57.649960 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.649808    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 16:19:57.649960 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.649824    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rpr2n\""
Sep 18 16:19:57.661020 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.660988    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8prt9"]
Sep 18 16:19:57.732855 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.732817    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.732855 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.732857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.733125 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.732980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2ad01b02-1165-4579-bfda-8fb5c6535ab7-crio-socket\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.733125 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.733055    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsjhp\" (UniqueName: \"kubernetes.io/projected/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-api-access-bsjhp\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.733125 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.733086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ad01b02-1165-4579-bfda-8fb5c6535ab7-data-volume\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.833494 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.833457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2ad01b02-1165-4579-bfda-8fb5c6535ab7-crio-socket\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.833666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.833521    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsjhp\" (UniqueName: \"kubernetes.io/projected/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-api-access-bsjhp\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.833666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.833550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ad01b02-1165-4579-bfda-8fb5c6535ab7-data-volume\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.833666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.833572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.833666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.833586    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2ad01b02-1165-4579-bfda-8fb5c6535ab7-crio-socket\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.833666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.833601    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.833666 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:57.833664    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:57.833931 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:57.833687    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:57.833931 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:57.833720    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:58.333705336 +0000 UTC m=+232.109137480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:57.833931 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:57.833734    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:58.333727177 +0000 UTC m=+232.109159317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:57.834556 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.834532    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2ad01b02-1165-4579-bfda-8fb5c6535ab7-data-volume\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:57.843075 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:57.843045    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsjhp\" (UniqueName: \"kubernetes.io/projected/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-api-access-bsjhp\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:58.338075 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.338048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:58.338519 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.338097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:58.338519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:58.338161    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:58.338519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:58.338195    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:58.338519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:58.338229    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:59.338212336 +0000 UTC m=+233.113644493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:58.338519 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:58.338268    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:19:59.33825266 +0000 UTC m=+233.113684813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:58.610195 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.610106    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"]
Sep 18 16:19:58.612854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.612832    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"
Sep 18 16:19:58.615769 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.615746    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rf9nh\""
Sep 18 16:19:58.615884 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.615765    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 16:19:58.616853 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.616836    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 16:19:58.621192 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.620970    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"]
Sep 18 16:19:58.639978 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.639941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njz5\" (UniqueName: \"kubernetes.io/projected/7bcd7861-01c5-4535-9a4b-bf6648d1ca52-kube-api-access-6njz5\") pod \"migrator-76794cbd6c-s2vjk\" (UID: \"7bcd7861-01c5-4535-9a4b-bf6648d1ca52\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"
Sep 18 16:19:58.740587 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.740544    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njz5\" (UniqueName: \"kubernetes.io/projected/7bcd7861-01c5-4535-9a4b-bf6648d1ca52-kube-api-access-6njz5\") pod \"migrator-76794cbd6c-s2vjk\" (UID: \"7bcd7861-01c5-4535-9a4b-bf6648d1ca52\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"
Sep 18 16:19:58.749500 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.749471    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njz5\" (UniqueName: \"kubernetes.io/projected/7bcd7861-01c5-4535-9a4b-bf6648d1ca52-kube-api-access-6njz5\") pod \"migrator-76794cbd6c-s2vjk\" (UID: \"7bcd7861-01c5-4535-9a4b-bf6648d1ca52\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"
Sep 18 16:19:58.922385 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:58.922356    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"
Sep 18 16:19:59.190128 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:59.190045    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk"]
Sep 18 16:19:59.193341 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:19:59.193300    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcd7861_01c5_4535_9a4b_bf6648d1ca52.slice/crio-1aa3365c26aeb16e441b0d08a37a900a90a16cd037271a3910d237a8c20dfafb WatchSource:0}: Error finding container 1aa3365c26aeb16e441b0d08a37a900a90a16cd037271a3910d237a8c20dfafb: Status 404 returned error can't find the container with id 1aa3365c26aeb16e441b0d08a37a900a90a16cd037271a3910d237a8c20dfafb
Sep 18 16:19:59.344304 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:59.344220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:59.344304 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:59.344290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:19:59.344829 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:59.344348    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:19:59.344829 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:59.344431    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.344410269 +0000 UTC m=+235.119842423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : configmap "kube-rbac-proxy" not found
Sep 18 16:19:59.344829 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:59.344462    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:59.344829 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:19:59.344541    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:01.34451217 +0000 UTC m=+235.119944310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:19:59.612855 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:19:59.612776    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pz2lq_ce80f8e8-94a6-4662-8531-c679fe377eef/dns-node-resolver/0.log"
Sep 18 16:20:00.199824 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:00.199787    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk" event={"ID":"7bcd7861-01c5-4535-9a4b-bf6648d1ca52","Type":"ContainerStarted","Data":"1aa3365c26aeb16e441b0d08a37a900a90a16cd037271a3910d237a8c20dfafb"}
Sep 18 16:20:00.413100 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:00.413068    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-csgxn_5e1d1ef2-8d2a-447b-a817-d4c67229a9db/node-ca/0.log"
Sep 18 16:20:01.204918 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.204882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk" event={"ID":"7bcd7861-01c5-4535-9a4b-bf6648d1ca52","Type":"ContainerStarted","Data":"e24d130db3e21fae5815e028b4bbf4e000b53027b24699c988216683ca1f7547"}
Sep 18 16:20:01.205067 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.204940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk" event={"ID":"7bcd7861-01c5-4535-9a4b-bf6648d1ca52","Type":"ContainerStarted","Data":"47b68ac3fbf57e5ac80c1c457dcb22f5cbcc13d8cae5a639874fe680ddf0067e"}
Sep 18 16:20:01.223189 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.223136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-s2vjk" podStartSLOduration=1.808103106 podStartE2EDuration="3.223116344s" totalImagesPullingTime="1.415013238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:01.222932162 +0000 UTC m=+234.998364350" watchObservedRunningTime="2026-09-18 16:20:01.223116344 +0000 UTC m=+234.998548507"
Sep 18 16:20:01.358716 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.358676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:01.358881 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.358729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:01.358881 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:01.358803    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:01.358881 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:01.358872    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:05.358856217 +0000 UTC m=+239.134288368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:01.359012 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:01.358891    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:01.359012 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:01.358940    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:05.358927693 +0000 UTC m=+239.134359837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:01.413197 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.413163    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-clq9q_4f877dfe-a591-4047-b728-942d3115b0ab/serve-healthcheck-canary/0.log"
Sep 18 16:20:01.459741 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.459657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:01.459741 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:01.459738    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:01.459903 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:01.459816    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:09.459799111 +0000 UTC m=+243.235231251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : configmap references non-existent config key: service-ca.crt
Sep 18 16:20:01.459903 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:01.459862    2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 18 16:20:01.459974 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:01.459906    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs podName:bedebae7-23bb-4bb2-8550-b66910de6e6d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:09.459894462 +0000 UTC m=+243.235326602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs") pod "router-default-656b55b86d-nx575" (UID: "bedebae7-23bb-4bb2-8550-b66910de6e6d") : secret "router-metrics-certs-default" not found
Sep 18 16:20:05.388957 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:05.388900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:05.389388 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:05.389010    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:05.389388 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:05.389060    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:05.389388 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:05.389077    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 16:20:05.389388 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:05.389129    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.389113167 +0000 UTC m=+247.164545307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : secret "insights-runtime-extractor-tls" not found
Sep 18 16:20:05.389388 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:05.389145    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm podName:2ad01b02-1165-4579-bfda-8fb5c6535ab7 nodeName:}" failed. No retries permitted until 2026-09-18 16:20:13.389138719 +0000 UTC m=+247.164570859 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8prt9" (UID: "2ad01b02-1165-4579-bfda-8fb5c6535ab7") : configmap "kube-rbac-proxy" not found
Sep 18 16:20:09.522620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:09.522576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:09.522990 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:09.522656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:09.523513 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:09.523494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bedebae7-23bb-4bb2-8550-b66910de6e6d-service-ca-bundle\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:09.525044 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:09.525020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bedebae7-23bb-4bb2-8550-b66910de6e6d-metrics-certs\") pod \"router-default-656b55b86d-nx575\" (UID: \"bedebae7-23bb-4bb2-8550-b66910de6e6d\") " pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:09.602498 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:09.602463    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fgfg6\""
Sep 18 16:20:09.610089 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:09.610061    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:09.882411 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:09.882378    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-656b55b86d-nx575"]
Sep 18 16:20:09.885377 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:09.885337    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedebae7_23bb_4bb2_8550_b66910de6e6d.slice/crio-69822e99174e43d58e1d1321bf465336842a484aa556f65a8d81b64a380a744f WatchSource:0}: Error finding container 69822e99174e43d58e1d1321bf465336842a484aa556f65a8d81b64a380a744f: Status 404 returned error can't find the container with id 69822e99174e43d58e1d1321bf465336842a484aa556f65a8d81b64a380a744f
Sep 18 16:20:10.232408 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:10.232321    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656b55b86d-nx575" event={"ID":"bedebae7-23bb-4bb2-8550-b66910de6e6d","Type":"ContainerStarted","Data":"417ccd904c3c46494be8268c04847eed5a7afa517b8aaea54420929c36fe5b84"}
Sep 18 16:20:10.232408 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:10.232376    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-656b55b86d-nx575" event={"ID":"bedebae7-23bb-4bb2-8550-b66910de6e6d","Type":"ContainerStarted","Data":"69822e99174e43d58e1d1321bf465336842a484aa556f65a8d81b64a380a744f"}
Sep 18 16:20:10.252414 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:10.252351    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-656b55b86d-nx575" podStartSLOduration=17.2523345 podStartE2EDuration="17.2523345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:10.252149086 +0000 UTC m=+244.027581248" watchObservedRunningTime="2026-09-18 16:20:10.2523345 +0000 UTC m=+244.027766659"
Sep 18 16:20:10.611332 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:10.611220    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:10.614475 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:10.614450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:11.235964 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:11.235927    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:11.237177 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:11.237158    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-656b55b86d-nx575"
Sep 18 16:20:13.452088 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:13.452052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:13.452088 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:13.452088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:13.452623 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:13.452601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2ad01b02-1165-4579-bfda-8fb5c6535ab7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:13.454404 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:13.454383    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2ad01b02-1165-4579-bfda-8fb5c6535ab7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8prt9\" (UID: \"2ad01b02-1165-4579-bfda-8fb5c6535ab7\") " pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:13.557579 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:13.557550    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rpr2n\""
Sep 18 16:20:13.566095 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:13.566056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8prt9"
Sep 18 16:20:13.851430 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:13.851355    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8prt9"]
Sep 18 16:20:13.854945 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:13.854917    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad01b02_1165_4579_bfda_8fb5c6535ab7.slice/crio-a7f73b63e40231d6b80ec833eec15909ee72ca09ee0c6052f02e953e556b6006 WatchSource:0}: Error finding container a7f73b63e40231d6b80ec833eec15909ee72ca09ee0c6052f02e953e556b6006: Status 404 returned error can't find the container with id a7f73b63e40231d6b80ec833eec15909ee72ca09ee0c6052f02e953e556b6006
Sep 18 16:20:14.247010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:14.246977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8prt9" event={"ID":"2ad01b02-1165-4579-bfda-8fb5c6535ab7","Type":"ContainerStarted","Data":"b5ecfc15a5cd63bb35bce813428b7b29817705052002f9f9bf4b6fbefc6a01af"}
Sep 18 16:20:14.247150 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:14.247033    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8prt9" event={"ID":"2ad01b02-1165-4579-bfda-8fb5c6535ab7","Type":"ContainerStarted","Data":"a7f73b63e40231d6b80ec833eec15909ee72ca09ee0c6052f02e953e556b6006"}
Sep 18 16:20:15.254874 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:15.254834    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8prt9" event={"ID":"2ad01b02-1165-4579-bfda-8fb5c6535ab7","Type":"ContainerStarted","Data":"5e93857df230534b9b13e483bf529d9200712e47101206fb6c55a3193f1b6cce"}
Sep 18 16:20:16.260369 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:16.260327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8prt9" event={"ID":"2ad01b02-1165-4579-bfda-8fb5c6535ab7","Type":"ContainerStarted","Data":"323512216c0b4f2c9facd6e3b1021a849f94a813c2db520dadd8f5203a1b7238"}
Sep 18 16:20:16.281309 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:16.281225    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8prt9" podStartSLOduration=17.42713007 podStartE2EDuration="19.28120746s" totalImagesPullingTime="1.85407739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 16:19:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:16.280480857 +0000 UTC m=+250.055913019" watchObservedRunningTime="2026-09-18 16:20:16.28120746 +0000 UTC m=+250.056639621"
Sep 18 16:20:19.097412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:19.097371    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:20:19.099658 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:19.099638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5d52384-daab-49e1-8414-893a899382b1-metrics-certs\") pod \"network-metrics-daemon-8g777\" (UID: \"c5d52384-daab-49e1-8414-893a899382b1\") " pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:20:19.231616 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:19.231538    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-px2dk\""
Sep 18 16:20:19.239780 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:19.239753    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8g777"
Sep 18 16:20:19.514746 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:19.514709    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8g777"]
Sep 18 16:20:19.517892 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:19.517862    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5d52384_daab_49e1_8414_893a899382b1.slice/crio-f23ee652672ab9d17a5619845cf767e150dcd869fe01777ee284822cbd920fcb WatchSource:0}: Error finding container f23ee652672ab9d17a5619845cf767e150dcd869fe01777ee284822cbd920fcb: Status 404 returned error can't find the container with id f23ee652672ab9d17a5619845cf767e150dcd869fe01777ee284822cbd920fcb
Sep 18 16:20:20.274813 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:20.274766    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8g777" event={"ID":"c5d52384-daab-49e1-8414-893a899382b1","Type":"ContainerStarted","Data":"f23ee652672ab9d17a5619845cf767e150dcd869fe01777ee284822cbd920fcb"}
Sep 18 16:20:21.279634 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:21.279597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8g777" event={"ID":"c5d52384-daab-49e1-8414-893a899382b1","Type":"ContainerStarted","Data":"b3dd9250be069894037ea21fe05e214a27550f1122a795e1e7a97ea41ecd028a"}
Sep 18 16:20:21.280113 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:21.279661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8g777" event={"ID":"c5d52384-daab-49e1-8414-893a899382b1","Type":"ContainerStarted","Data":"e2f59d79ebdceed5283bbbe8afc05fc7cb6eccdf0449922cb1fe3ef1f2d95e25"}
Sep 18 16:20:21.297673 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:21.297608    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8g777" podStartSLOduration=252.298235355 podStartE2EDuration="4m13.297587151s" totalImagesPullingTime="999.351796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:21.296323943 +0000 UTC m=+255.071756126" watchObservedRunningTime="2026-09-18 16:20:21.297587151 +0000 UTC m=+255.073019314"
Sep 18 16:20:28.324693 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.324652    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"]
Sep 18 16:20:28.330017 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.329990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"
Sep 18 16:20:28.334424 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.334391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5qqd2\""
Sep 18 16:20:28.334596 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.334392    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 18 16:20:28.346575 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.346546    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"]
Sep 18 16:20:28.470159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.470106    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adeb27ae-270f-4d03-99b0-eeb07bc30b3f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jrjjk\" (UID: \"adeb27ae-270f-4d03-99b0-eeb07bc30b3f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"
Sep 18 16:20:28.571412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.571378    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adeb27ae-270f-4d03-99b0-eeb07bc30b3f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jrjjk\" (UID: \"adeb27ae-270f-4d03-99b0-eeb07bc30b3f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"
Sep 18 16:20:28.573801 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.573780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/adeb27ae-270f-4d03-99b0-eeb07bc30b3f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-jrjjk\" (UID: \"adeb27ae-270f-4d03-99b0-eeb07bc30b3f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"
Sep 18 16:20:28.645381 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.645343    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"
Sep 18 16:20:28.916012 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:28.915924    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"]
Sep 18 16:20:28.918965 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:28.918939    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadeb27ae_270f_4d03_99b0_eeb07bc30b3f.slice/crio-9f4b35af7e24091a106b23b66eaaa35a80426bd60c1e6b7b871f534169e30649 WatchSource:0}: Error finding container 9f4b35af7e24091a106b23b66eaaa35a80426bd60c1e6b7b871f534169e30649: Status 404 returned error can't find the container with id 9f4b35af7e24091a106b23b66eaaa35a80426bd60c1e6b7b871f534169e30649
Sep 18 16:20:29.305888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:29.305799    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk" event={"ID":"adeb27ae-270f-4d03-99b0-eeb07bc30b3f","Type":"ContainerStarted","Data":"9f4b35af7e24091a106b23b66eaaa35a80426bd60c1e6b7b871f534169e30649"}
Sep 18 16:20:30.310658 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:30.310571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk" event={"ID":"adeb27ae-270f-4d03-99b0-eeb07bc30b3f","Type":"ContainerStarted","Data":"ea8931a9244abc5ab2a0668a236356d722d77075bc5c82b75d9900b2803071e7"}
Sep 18 16:20:30.311069 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:30.310771    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"
Sep 18 16:20:30.315406 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:30.315380    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk"
Sep 18 16:20:30.327652 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:30.327599    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-jrjjk" podStartSLOduration=1.220386335 podStartE2EDuration="2.327583742s" totalImagesPullingTime="1.107197407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:30.327147987 +0000 UTC m=+264.102580175" watchObservedRunningTime="2026-09-18 16:20:30.327583742 +0000 UTC m=+264.103015903"
Sep 18 16:20:35.767329 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.767290    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"]
Sep 18 16:20:35.770365 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.770343    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:35.774516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.774332    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 16:20:35.774516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.774359    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 18 16:20:35.774516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.774359    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 16:20:35.774516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.774405    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 18 16:20:35.774516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.774346    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s9zmh\""
Sep 18 16:20:35.774516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.774450    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 16:20:35.774812 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.774679    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 18 16:20:35.775649 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.775624    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fhlhc"]
Sep 18 16:20:35.778148 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.778130    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.782333 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.782316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 16:20:35.782659 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.782644    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 16:20:35.784465 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.784263    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 16:20:35.784465 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.784429    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-57lgg\""
Sep 18 16:20:35.787641 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.787592    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"]
Sep 18 16:20:35.923414 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:35.923414 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923415    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a69f4a3-abae-4da2-957f-c10b107872b9-metrics-client-ca\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923651 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923446    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923651 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923651 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923515    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-sys\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923651 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:35.923651 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/04bc1f71-dacf-4cdf-b84f-de5d1519911d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:35.923651 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-tls\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmbx9\" (UniqueName: \"kubernetes.io/projected/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-api-access-pmbx9\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:35.923854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:35.923854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-textfile\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923780    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-wtmp\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923806    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-root\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:35.923854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bc1f71-dacf-4cdf-b84f-de5d1519911d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:35.924046 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:35.923854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6hw\" (UniqueName: \"kubernetes.io/projected/9a69f4a3-abae-4da2-957f-c10b107872b9-kube-api-access-jn6hw\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025006 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.024902    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/04bc1f71-dacf-4cdf-b84f-de5d1519911d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.025006 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.024953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-tls\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025006 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.024981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmbx9\" (UniqueName: \"kubernetes.io/projected/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-api-access-pmbx9\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-textfile\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-wtmp\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-root\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025150    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bc1f71-dacf-4cdf-b84f-de5d1519911d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025178    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6hw\" (UniqueName: \"kubernetes.io/projected/9a69f4a3-abae-4da2-957f-c10b107872b9-kube-api-access-jn6hw\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025218    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.025316 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a69f4a3-abae-4da2-957f-c10b107872b9-metrics-client-ca\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-wtmp\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025389    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/04bc1f71-dacf-4cdf-b84f-de5d1519911d-volume-directive-shadow\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025408    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-root\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-sys\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a69f4a3-abae-4da2-957f-c10b107872b9-sys\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:36.025499    2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025498    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-textfile\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:36.025566    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-tls podName:04bc1f71-dacf-4cdf-b84f-de5d1519911d nodeName:}" failed. No retries permitted until 2026-09-18 16:20:36.525547283 +0000 UTC m=+270.300979433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-tls") pod "kube-state-metrics-84c885658c-r8zp5" (UID: "04bc1f71-dacf-4cdf-b84f-de5d1519911d") : secret "kube-state-metrics-tls" not found
Sep 18 16:20:36.025692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.026337 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.025958    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a69f4a3-abae-4da2-957f-c10b107872b9-metrics-client-ca\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.026337 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.026068    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04bc1f71-dacf-4cdf-b84f-de5d1519911d-metrics-client-ca\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.026337 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.026289    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.026492 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.026436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.027792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.027768    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.027961 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.027944    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.028047 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.028031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a69f4a3-abae-4da2-957f-c10b107872b9-node-exporter-tls\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.036207 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.036184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmbx9\" (UniqueName: \"kubernetes.io/projected/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-api-access-pmbx9\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.036427 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.036405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6hw\" (UniqueName: \"kubernetes.io/projected/9a69f4a3-abae-4da2-957f-c10b107872b9-kube-api-access-jn6hw\") pod \"node-exporter-fhlhc\" (UID: \"9a69f4a3-abae-4da2-957f-c10b107872b9\") " pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.087061 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.087028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fhlhc"
Sep 18 16:20:36.097742 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:36.097709    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a69f4a3_abae_4da2_957f_c10b107872b9.slice/crio-3f4a945a3082632c9a2821a38425c0f157e17d47e2109e6d7386e2d597c33c86 WatchSource:0}: Error finding container 3f4a945a3082632c9a2821a38425c0f157e17d47e2109e6d7386e2d597c33c86: Status 404 returned error can't find the container with id 3f4a945a3082632c9a2821a38425c0f157e17d47e2109e6d7386e2d597c33c86
Sep 18 16:20:36.332834 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.332750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhlhc" event={"ID":"9a69f4a3-abae-4da2-957f-c10b107872b9","Type":"ContainerStarted","Data":"3f4a945a3082632c9a2821a38425c0f157e17d47e2109e6d7386e2d597c33c86"}
Sep 18 16:20:36.529360 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.529317    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.531682 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.531663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/04bc1f71-dacf-4cdf-b84f-de5d1519911d-kube-state-metrics-tls\") pod \"kube-state-metrics-84c885658c-r8zp5\" (UID: \"04bc1f71-dacf-4cdf-b84f-de5d1519911d\") " pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.680609 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.680571    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"
Sep 18 16:20:36.954821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:36.954730    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-84c885658c-r8zp5"]
Sep 18 16:20:37.151461 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:37.151421    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bc1f71_dacf_4cdf_b84f_de5d1519911d.slice/crio-71214bca450852b5b581bc4ab216673ecd81cdd6d1b98668a69ec99f7beee482 WatchSource:0}: Error finding container 71214bca450852b5b581bc4ab216673ecd81cdd6d1b98668a69ec99f7beee482: Status 404 returned error can't find the container with id 71214bca450852b5b581bc4ab216673ecd81cdd6d1b98668a69ec99f7beee482
Sep 18 16:20:37.339943 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:37.339901    2562 generic.go:356] "Generic (PLEG): container finished" podID="9a69f4a3-abae-4da2-957f-c10b107872b9" containerID="0ec01f3dd3f7e74b65b49a71211c2ebc4903e796d8f56c6354d642622f5d0fc2" exitCode=0
Sep 18 16:20:37.340130 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:37.339983    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhlhc" event={"ID":"9a69f4a3-abae-4da2-957f-c10b107872b9","Type":"ContainerDied","Data":"0ec01f3dd3f7e74b65b49a71211c2ebc4903e796d8f56c6354d642622f5d0fc2"}
Sep 18 16:20:37.341284 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:37.341263    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5" event={"ID":"04bc1f71-dacf-4cdf-b84f-de5d1519911d","Type":"ContainerStarted","Data":"71214bca450852b5b581bc4ab216673ecd81cdd6d1b98668a69ec99f7beee482"}
Sep 18 16:20:38.347616 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:38.347582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhlhc" event={"ID":"9a69f4a3-abae-4da2-957f-c10b107872b9","Type":"ContainerStarted","Data":"6732d55d1a09f19fa4db5b7c4a88091081138c831312b2caa1038cd528f8864d"}
Sep 18 16:20:38.348033 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:38.347648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fhlhc" event={"ID":"9a69f4a3-abae-4da2-957f-c10b107872b9","Type":"ContainerStarted","Data":"e3b149abf12c962f7189717a1fba885c17d64c02e72b23b8eeafd347c2a32fe7"}
Sep 18 16:20:38.349873 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:38.349845    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5" event={"ID":"04bc1f71-dacf-4cdf-b84f-de5d1519911d","Type":"ContainerStarted","Data":"e267f54378f88fd52984aaa79ee172a511af1e26472f39e3b03373e8813b611f"}
Sep 18 16:20:38.349993 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:38.349897    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5" event={"ID":"04bc1f71-dacf-4cdf-b84f-de5d1519911d","Type":"ContainerStarted","Data":"95d2ca611cd47d3a6dd6514873eef224c82497c462bab9f68712d5fb19c31c20"}
Sep 18 16:20:38.372415 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:38.372364    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fhlhc" podStartSLOduration=2.2284248939999998 podStartE2EDuration="3.37234588s" totalImagesPullingTime="1.079208121s" totalInitContainerRuntime="64.712865ms" isStatefulPod=true podCreationTimestamp="2026-09-18 16:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:38.370363025 +0000 UTC m=+272.145795212" watchObservedRunningTime="2026-09-18 16:20:38.37234588 +0000 UTC m=+272.147778042"
Sep 18 16:20:39.355605 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:39.355569    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5" event={"ID":"04bc1f71-dacf-4cdf-b84f-de5d1519911d","Type":"ContainerStarted","Data":"ba152f23ceb6b9fbc9aa063a300842ca44fb3859e4cf5cd8e910ad55a4e50e9f"}
Sep 18 16:20:39.379760 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:39.379706    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-84c885658c-r8zp5" podStartSLOduration=3.359153425 podStartE2EDuration="4.379687109s" totalImagesPullingTime="1.020533684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:39.378805211 +0000 UTC m=+273.154237372" watchObservedRunningTime="2026-09-18 16:20:39.379687109 +0000 UTC m=+273.155119308"
Sep 18 16:20:40.496390 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.496349    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"]
Sep 18 16:20:40.498797 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.498781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:40.501198 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.501175    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 18 16:20:40.501322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.501255    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vjgsk\""
Sep 18 16:20:40.509630 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.509601    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"]
Sep 18 16:20:40.663542 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.663503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvqjv\" (UniqueName: \"kubernetes.io/projected/0a7469ca-1aa3-4484-9ef4-0fd40e7174dc-kube-api-access-cvqjv\") pod \"monitoring-plugin-5984459ff8-wmgqp\" (UID: \"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:40.663692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.663615    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a7469ca-1aa3-4484-9ef4-0fd40e7174dc-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-wmgqp\" (UID: \"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:40.764335 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.764185    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a7469ca-1aa3-4484-9ef4-0fd40e7174dc-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-wmgqp\" (UID: \"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:40.764335 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.764309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvqjv\" (UniqueName: \"kubernetes.io/projected/0a7469ca-1aa3-4484-9ef4-0fd40e7174dc-kube-api-access-cvqjv\") pod \"monitoring-plugin-5984459ff8-wmgqp\" (UID: \"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:40.766855 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.766832    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0a7469ca-1aa3-4484-9ef4-0fd40e7174dc-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-wmgqp\" (UID: \"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:40.773594 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.773568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvqjv\" (UniqueName: \"kubernetes.io/projected/0a7469ca-1aa3-4484-9ef4-0fd40e7174dc-kube-api-access-cvqjv\") pod \"monitoring-plugin-5984459ff8-wmgqp\" (UID: \"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:40.808564 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:40.808528    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:41.096281 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:41.096179    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"]
Sep 18 16:20:41.099211 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:41.099183    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7469ca_1aa3_4484_9ef4_0fd40e7174dc.slice/crio-fb8f89285bf052dbde73ab5f11249f322fe3d7b0742bca9ed42b99965596c3ad WatchSource:0}: Error finding container fb8f89285bf052dbde73ab5f11249f322fe3d7b0742bca9ed42b99965596c3ad: Status 404 returned error can't find the container with id fb8f89285bf052dbde73ab5f11249f322fe3d7b0742bca9ed42b99965596c3ad
Sep 18 16:20:41.364951 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:41.364918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp" event={"ID":"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc","Type":"ContainerStarted","Data":"fb8f89285bf052dbde73ab5f11249f322fe3d7b0742bca9ed42b99965596c3ad"}
Sep 18 16:20:43.373974 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:43.373935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp" event={"ID":"0a7469ca-1aa3-4484-9ef4-0fd40e7174dc","Type":"ContainerStarted","Data":"6aa18102fd8b51acd5955fe5b9bd340bfa6a70e7936fa00ac6179aff5e261c53"}
Sep 18 16:20:43.374406 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:43.374125    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:43.378962 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:43.378936    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp"
Sep 18 16:20:43.394487 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:43.394427    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-wmgqp" podStartSLOduration=1.937702356 podStartE2EDuration="3.394408533s" totalImagesPullingTime="1.456706177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:43.39248831 +0000 UTC m=+277.167920501" watchObservedRunningTime="2026-09-18 16:20:43.394408533 +0000 UTC m=+277.169840694"
Sep 18 16:20:49.003278 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:20:49.003221    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-km6fd" podUID="a7d64b9b-0a6d-404b-9f0a-317263c530f4"
Sep 18 16:20:49.395438 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:49.395411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-km6fd"
Sep 18 16:20:51.950402 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:51.948306    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-tlpft"]
Sep 18 16:20:51.953072 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:51.951722    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-tlpft"
Sep 18 16:20:51.954822 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:51.954791    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 16:20:51.954957 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:51.954823    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h54pq\""
Sep 18 16:20:51.954957 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:51.954836    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 16:20:51.961004 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:51.960977    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-tlpft"]
Sep 18 16:20:52.049734 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.049691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rzq\" (UniqueName: \"kubernetes.io/projected/aa146f83-a717-4a52-a2e8-fe3c5ccb42d2-kube-api-access-75rzq\") pod \"downloads-6594f4c75-tlpft\" (UID: \"aa146f83-a717-4a52-a2e8-fe3c5ccb42d2\") " pod="openshift-console/downloads-6594f4c75-tlpft"
Sep 18 16:20:52.150183 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.150146    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rzq\" (UniqueName: \"kubernetes.io/projected/aa146f83-a717-4a52-a2e8-fe3c5ccb42d2-kube-api-access-75rzq\") pod \"downloads-6594f4c75-tlpft\" (UID: \"aa146f83-a717-4a52-a2e8-fe3c5ccb42d2\") " pod="openshift-console/downloads-6594f4c75-tlpft"
Sep 18 16:20:52.161384 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.161347    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rzq\" (UniqueName: \"kubernetes.io/projected/aa146f83-a717-4a52-a2e8-fe3c5ccb42d2-kube-api-access-75rzq\") pod \"downloads-6594f4c75-tlpft\" (UID: \"aa146f83-a717-4a52-a2e8-fe3c5ccb42d2\") " pod="openshift-console/downloads-6594f4c75-tlpft"
Sep 18 16:20:52.263484 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.263402    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-tlpft"
Sep 18 16:20:52.539439 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.539411    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-tlpft"]
Sep 18 16:20:52.542368 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:52.542329    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa146f83_a717_4a52_a2e8_fe3c5ccb42d2.slice/crio-59b7e40835421297e49d7972bf24f8a188774800f422269ce13254eba3f2b9c3 WatchSource:0}: Error finding container 59b7e40835421297e49d7972bf24f8a188774800f422269ce13254eba3f2b9c3: Status 404 returned error can't find the container with id 59b7e40835421297e49d7972bf24f8a188774800f422269ce13254eba3f2b9c3
Sep 18 16:20:52.553844 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.553811    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:20:52.556292 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.556266    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a7d64b9b-0a6d-404b-9f0a-317263c530f4-metrics-tls\") pod \"dns-default-km6fd\" (UID: \"a7d64b9b-0a6d-404b-9f0a-317263c530f4\") " pod="openshift-dns/dns-default-km6fd"
Sep 18 16:20:52.699931 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.699892    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-f68fb\""
Sep 18 16:20:52.707213 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.707186    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-km6fd"
Sep 18 16:20:52.986698 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:52.986661    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-km6fd"]
Sep 18 16:20:52.989839 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:20:52.989807    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d64b9b_0a6d_404b_9f0a_317263c530f4.slice/crio-1ac8a0e6c64ab6bd5b0da1f21bf372d4f03409b265dd63a17cf0cff265cd7335 WatchSource:0}: Error finding container 1ac8a0e6c64ab6bd5b0da1f21bf372d4f03409b265dd63a17cf0cff265cd7335: Status 404 returned error can't find the container with id 1ac8a0e6c64ab6bd5b0da1f21bf372d4f03409b265dd63a17cf0cff265cd7335
Sep 18 16:20:53.411792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:53.411750    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-km6fd" event={"ID":"a7d64b9b-0a6d-404b-9f0a-317263c530f4","Type":"ContainerStarted","Data":"1ac8a0e6c64ab6bd5b0da1f21bf372d4f03409b265dd63a17cf0cff265cd7335"}
Sep 18 16:20:53.413505 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:53.413466    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-tlpft" event={"ID":"aa146f83-a717-4a52-a2e8-fe3c5ccb42d2","Type":"ContainerStarted","Data":"59b7e40835421297e49d7972bf24f8a188774800f422269ce13254eba3f2b9c3"}
Sep 18 16:20:55.425750 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:55.425710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-km6fd" event={"ID":"a7d64b9b-0a6d-404b-9f0a-317263c530f4","Type":"ContainerStarted","Data":"2da78b3ae51e0fa3715207373b84efb0ad8799f5eb3a9e7afece02fb183f8d52"}
Sep 18 16:20:55.426229 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:55.425777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-km6fd" event={"ID":"a7d64b9b-0a6d-404b-9f0a-317263c530f4","Type":"ContainerStarted","Data":"fb8a791a18f04933e14b89f34572e9c3a73793afd29533d2d582732f6987c586"}
Sep 18 16:20:55.426229 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:55.425867    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-km6fd"
Sep 18 16:20:55.445683 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:20:55.445628    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-km6fd" podStartSLOduration=251.704818061 podStartE2EDuration="4m13.445611324s" totalImagesPullingTime="1.740793263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:20:55.442887325 +0000 UTC m=+289.218319511" watchObservedRunningTime="2026-09-18 16:20:55.445611324 +0000 UTC m=+289.221043486"
Sep 18 16:21:05.431949 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:05.431901    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-km6fd"
Sep 18 16:21:09.952752 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:09.952725    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 16:21:10.493251 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:10.493182    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-tlpft" event={"ID":"aa146f83-a717-4a52-a2e8-fe3c5ccb42d2","Type":"ContainerStarted","Data":"02d410c2a47345d67139d81dcd1b6c70ebe8b3ed7078dc804bd3ebccd54e579e"}
Sep 18 16:21:10.493456 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:10.493393    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-tlpft"
Sep 18 16:21:10.512852 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:10.512789    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-tlpft" podStartSLOduration=2.039267399 podStartE2EDuration="19.51277063s" totalImagesPullingTime="17.473503231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:20:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:21:10.511814362 +0000 UTC m=+304.287246542" watchObservedRunningTime="2026-09-18 16:21:10.51277063 +0000 UTC m=+304.288202823"
Sep 18 16:21:10.516137 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:10.516105    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-tlpft"
Sep 18 16:21:24.548223 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:24.548193    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-r8zp5_04bc1f71-dacf-4cdf-b84f-de5d1519911d/kube-state-metrics/0.log"
Sep 18 16:21:24.748392 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:24.748362    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-r8zp5_04bc1f71-dacf-4cdf-b84f-de5d1519911d/kube-rbac-proxy-main/0.log"
Sep 18 16:21:24.948396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:24.948368    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-r8zp5_04bc1f71-dacf-4cdf-b84f-de5d1519911d/kube-rbac-proxy-self/0.log"
Sep 18 16:21:25.347775 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:25.347688    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-wmgqp_0a7469ca-1aa3-4484-9ef4-0fd40e7174dc/monitoring-plugin/0.log"
Sep 18 16:21:26.148202 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:26.148175    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhlhc_9a69f4a3-abae-4da2-957f-c10b107872b9/init-textfile/0.log"
Sep 18 16:21:26.348175 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:26.348139    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhlhc_9a69f4a3-abae-4da2-957f-c10b107872b9/node-exporter/0.log"
Sep 18 16:21:26.548799 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:26.548697    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhlhc_9a69f4a3-abae-4da2-957f-c10b107872b9/kube-rbac-proxy/0.log"
Sep 18 16:21:27.560439 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:27.560404    2562 generic.go:356] "Generic (PLEG): container finished" podID="362c0f6a-10ae-45e0-bcc4-ecddfbea6a64" containerID="20b84da5f214766eb81e8004fdeacec4f3b0c554ccf1751ea786e94b19fcfc51" exitCode=0
Sep 18 16:21:27.560823 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:27.560468    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld" event={"ID":"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64","Type":"ContainerDied","Data":"20b84da5f214766eb81e8004fdeacec4f3b0c554ccf1751ea786e94b19fcfc51"}
Sep 18 16:21:27.560823 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:27.560751    2562 scope.go:117] "RemoveContainer" containerID="20b84da5f214766eb81e8004fdeacec4f3b0c554ccf1751ea786e94b19fcfc51"
Sep 18 16:21:28.347674 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:28.347645    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-jrjjk_adeb27ae-270f-4d03-99b0-eeb07bc30b3f/prometheus-operator-admission-webhook/0.log"
Sep 18 16:21:28.566047 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:28.566005    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-55889f7b47-l4lld" event={"ID":"362c0f6a-10ae-45e0-bcc4-ecddfbea6a64","Type":"ContainerStarted","Data":"d191a5e98a5b88cfef359503e8517bbf7b68979cfa512e962c636fc4b13e36f3"}
Sep 18 16:21:30.149879 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:30.149849    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-tlpft_aa146f83-a717-4a52-a2e8-fe3c5ccb42d2/download-server/0.log"
Sep 18 16:21:30.348354 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:30.348327    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656b55b86d-nx575_bedebae7-23bb-4bb2-8550-b66910de6e6d/router/0.log"
Sep 18 16:21:30.748536 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:21:30.748508    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-clq9q_4f877dfe-a591-4047-b728-942d3115b0ab/serve-healthcheck-canary/0.log"
Sep 18 16:25:40.960680 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.960626    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67d9498446-pl6zs"]
Sep 18 16:25:40.963377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.963354    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:40.966219 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.966188    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 18 16:25:40.966386 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.966345    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 18 16:25:40.966441 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.966381    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bl2m4ttst0r69\""
Sep 18 16:25:40.966441 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.966404    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rr6pk\""
Sep 18 16:25:40.966535 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.966514    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 18 16:25:40.966599 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.966584    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 18 16:25:40.967125 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.967106    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 18 16:25:40.977028 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:40.977003    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67d9498446-pl6zs"]
Sep 18 16:25:41.003646 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.003646 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003647    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.003861 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003697    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6412340b-599d-4df9-80ab-51cebf8702fe-metrics-client-ca\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.003861 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.003861 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb8cq\" (UniqueName: \"kubernetes.io/projected/6412340b-599d-4df9-80ab-51cebf8702fe-kube-api-access-kb8cq\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.003861 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.003861 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-tls\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.004019 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.003876    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-grpc-tls\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.104740 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.104699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.104740 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.104742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.104918 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.104871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6412340b-599d-4df9-80ab-51cebf8702fe-metrics-client-ca\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.104918 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.104912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.105138 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.105115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb8cq\" (UniqueName: \"kubernetes.io/projected/6412340b-599d-4df9-80ab-51cebf8702fe-kube-api-access-kb8cq\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.105215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.105164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.105297 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.105272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-tls\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.105350 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.105308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-grpc-tls\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.105689 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.105661    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6412340b-599d-4df9-80ab-51cebf8702fe-metrics-client-ca\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.107572 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.107541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.107860 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.107830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.107966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.107866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.108029 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.107970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-tls\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.108089 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.108066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.108251 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.108222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6412340b-599d-4df9-80ab-51cebf8702fe-secret-grpc-tls\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.113839 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.113812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb8cq\" (UniqueName: \"kubernetes.io/projected/6412340b-599d-4df9-80ab-51cebf8702fe-kube-api-access-kb8cq\") pod \"thanos-querier-67d9498446-pl6zs\" (UID: \"6412340b-599d-4df9-80ab-51cebf8702fe\") " pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.273175 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.273078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:41.553338 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.553314    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67d9498446-pl6zs"]
Sep 18 16:25:41.555066 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:25:41.555033    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6412340b_599d_4df9_80ab_51cebf8702fe.slice/crio-474177c3ae02b4ac6dc9eda1737ed83df9a7a155d5125483f8f9201a93897556 WatchSource:0}: Error finding container 474177c3ae02b4ac6dc9eda1737ed83df9a7a155d5125483f8f9201a93897556: Status 404 returned error can't find the container with id 474177c3ae02b4ac6dc9eda1737ed83df9a7a155d5125483f8f9201a93897556
Sep 18 16:25:41.557207 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:41.557191    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:25:42.496295 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:42.496252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" event={"ID":"6412340b-599d-4df9-80ab-51cebf8702fe","Type":"ContainerStarted","Data":"474177c3ae02b4ac6dc9eda1737ed83df9a7a155d5125483f8f9201a93897556"}
Sep 18 16:25:43.502828 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:43.502791    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" event={"ID":"6412340b-599d-4df9-80ab-51cebf8702fe","Type":"ContainerStarted","Data":"2eddf3a4e9f59f2f5d919cc8a38a682e643d8df264b2a1cc89da5626275b6ce7"}
Sep 18 16:25:43.503217 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:43.502859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" event={"ID":"6412340b-599d-4df9-80ab-51cebf8702fe","Type":"ContainerStarted","Data":"bb20e53da8a6f538ad95de45bce712342340d46ddcb302a285654ac03a05156b"}
Sep 18 16:25:43.503217 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:43.502888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" event={"ID":"6412340b-599d-4df9-80ab-51cebf8702fe","Type":"ContainerStarted","Data":"9200cd1260a4dadd54ccdd5c3eca7cfe77f0ad328f979ad862e04a552ce49fb7"}
Sep 18 16:25:44.510992 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:44.510955    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" event={"ID":"6412340b-599d-4df9-80ab-51cebf8702fe","Type":"ContainerStarted","Data":"5c222731613ae2bab197963d4234f4098acc8faba2426e1e212bc32185409c07"}
Sep 18 16:25:44.511428 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:44.511044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" event={"ID":"6412340b-599d-4df9-80ab-51cebf8702fe","Type":"ContainerStarted","Data":"f53a35ece14a1dd645fdb6562e232ded5db66c1bfb5c83fe31c6cbd0e7e80018"}
Sep 18 16:25:45.518963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:45.518926    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" event={"ID":"6412340b-599d-4df9-80ab-51cebf8702fe","Type":"ContainerStarted","Data":"b8b53aa04259bc4a016d2909e2bf54c0c93818ab504a8e4a6247794446debffa"}
Sep 18 16:25:45.519468 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:45.519048    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:25:45.544792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:45.544736    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs" podStartSLOduration=2.9006798849999997 podStartE2EDuration="5.544718931s" totalImagesPullingTime="2.644039046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:25:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:25:45.543180043 +0000 UTC m=+579.318612232" watchObservedRunningTime="2026-09-18 16:25:45.544718931 +0000 UTC m=+579.320151090"
Sep 18 16:25:51.529071 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:25:51.529038    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67d9498446-pl6zs"
Sep 18 16:27:47.487469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.487426    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"]
Sep 18 16:27:47.490732 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.490713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.495195 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.495173    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 16:27:47.495336 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.495252    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 16:27:47.495336 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.495284    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 16:27:47.495336 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.495309    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 16:27:47.503507 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.503481    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"]
Sep 18 16:27:47.540323 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.540279    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7e1c889-ce76-4464-8751-281f42e20d3c-klusterlet-config\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.540514 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.540394    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jxfr\" (UniqueName: \"kubernetes.io/projected/d7e1c889-ce76-4464-8751-281f42e20d3c-kube-api-access-9jxfr\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.540514 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.540431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e1c889-ce76-4464-8751-281f42e20d3c-tmp\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.641586 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.641547    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jxfr\" (UniqueName: \"kubernetes.io/projected/d7e1c889-ce76-4464-8751-281f42e20d3c-kube-api-access-9jxfr\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.641586 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.641585    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e1c889-ce76-4464-8751-281f42e20d3c-tmp\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.641857 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.641603    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7e1c889-ce76-4464-8751-281f42e20d3c-klusterlet-config\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.641983 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.641961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d7e1c889-ce76-4464-8751-281f42e20d3c-tmp\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.644090 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.644061    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d7e1c889-ce76-4464-8751-281f42e20d3c-klusterlet-config\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.649813 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.649788    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jxfr\" (UniqueName: \"kubernetes.io/projected/d7e1c889-ce76-4464-8751-281f42e20d3c-kube-api-access-9jxfr\") pod \"klusterlet-addon-workmgr-66ff4fc578-2lf84\" (UID: \"d7e1c889-ce76-4464-8751-281f42e20d3c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:47.799863 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:47.799764    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:48.069889 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:48.069801    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"]
Sep 18 16:27:48.072786 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:27:48.072757    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e1c889_ce76_4464_8751_281f42e20d3c.slice/crio-89339bd12315bcf6dba442cb4f980085fb8c331dd86ccd957b5168f92551b4f1 WatchSource:0}: Error finding container 89339bd12315bcf6dba442cb4f980085fb8c331dd86ccd957b5168f92551b4f1: Status 404 returned error can't find the container with id 89339bd12315bcf6dba442cb4f980085fb8c331dd86ccd957b5168f92551b4f1
Sep 18 16:27:48.977876 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:48.977836    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84" event={"ID":"d7e1c889-ce76-4464-8751-281f42e20d3c","Type":"ContainerStarted","Data":"89339bd12315bcf6dba442cb4f980085fb8c331dd86ccd957b5168f92551b4f1"}
Sep 18 16:27:52.000627 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:52.000207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84" event={"ID":"d7e1c889-ce76-4464-8751-281f42e20d3c","Type":"ContainerStarted","Data":"583a9263417c522f301b7ba246ed67df2eda7e563717ad54a385c3eecba24198"}
Sep 18 16:27:52.000627 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:52.000576    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:27:52.002160 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:52.002122    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84" podUID="d7e1c889-ce76-4464-8751-281f42e20d3c" containerName="acm-agent" probeResult="failure" output="Get \"http://10.134.0.19:8000/readyz\": dial tcp 10.134.0.19:8000: connect: connection refused"
Sep 18 16:27:52.017613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:52.017451    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84" podStartSLOduration=1.177039421 podStartE2EDuration="5.017431813s" totalImagesPullingTime="3.840392392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:27:52.017045153 +0000 UTC m=+705.792477338" watchObservedRunningTime="2026-09-18 16:27:52.017431813 +0000 UTC m=+705.792863975"
Sep 18 16:27:53.005790 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:27:53.005759    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66ff4fc578-2lf84"
Sep 18 16:29:43.347581 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.347535    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"]
Sep 18 16:29:43.351214 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.351185    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.356205 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.356177    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jv47d\""
Sep 18 16:29:43.356384 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.356298    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 16:29:43.357282 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.357256    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 16:29:43.387228 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.387195    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"]
Sep 18 16:29:43.438189 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.438153    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.438189 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.438196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.438439 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.438299    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db799\" (UniqueName: \"kubernetes.io/projected/b87c925c-c5f6-4b70-97c9-b795c672c0cc-kube-api-access-db799\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.539478 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.539428    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.539676 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.539488    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.539676 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.539540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db799\" (UniqueName: \"kubernetes.io/projected/b87c925c-c5f6-4b70-97c9-b795c672c0cc-kube-api-access-db799\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.539835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.539814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.539908 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.539844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.548831 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.548801    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db799\" (UniqueName: \"kubernetes.io/projected/b87c925c-c5f6-4b70-97c9-b795c672c0cc-kube-api-access-db799\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.661340 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.661301    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:29:43.938390 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:43.938300    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"]
Sep 18 16:29:43.941523 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:29:43.941492    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87c925c_c5f6_4b70_97c9_b795c672c0cc.slice/crio-d8cb3dfeb2908b55a53ed753604158ac1f6ff77e977bef2c115f34578bb984ba WatchSource:0}: Error finding container d8cb3dfeb2908b55a53ed753604158ac1f6ff77e977bef2c115f34578bb984ba: Status 404 returned error can't find the container with id d8cb3dfeb2908b55a53ed753604158ac1f6ff77e977bef2c115f34578bb984ba
Sep 18 16:29:44.426635 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:44.426598    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95" event={"ID":"b87c925c-c5f6-4b70-97c9-b795c672c0cc","Type":"ContainerStarted","Data":"d8cb3dfeb2908b55a53ed753604158ac1f6ff77e977bef2c115f34578bb984ba"}
Sep 18 16:29:50.455100 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:50.455062    2562 generic.go:356] "Generic (PLEG): container finished" podID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerID="4ba3a63d808f82d5b2335e9edf72282f0a272acf8d79c2174cb01f0cd66a5f09" exitCode=0
Sep 18 16:29:50.455583 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:50.455150    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95" event={"ID":"b87c925c-c5f6-4b70-97c9-b795c672c0cc","Type":"ContainerDied","Data":"4ba3a63d808f82d5b2335e9edf72282f0a272acf8d79c2174cb01f0cd66a5f09"}
Sep 18 16:29:53.471337 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:53.471300    2562 generic.go:356] "Generic (PLEG): container finished" podID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerID="e71ad618651ff7681fcbd445fa8712175a61818e7d63d3576b4a7a6d3896d77b" exitCode=0
Sep 18 16:29:53.471735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:29:53.471381    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95" event={"ID":"b87c925c-c5f6-4b70-97c9-b795c672c0cc","Type":"ContainerDied","Data":"e71ad618651ff7681fcbd445fa8712175a61818e7d63d3576b4a7a6d3896d77b"}
Sep 18 16:30:00.502903 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:00.502866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95" event={"ID":"b87c925c-c5f6-4b70-97c9-b795c672c0cc","Type":"ContainerStarted","Data":"2bb79f06c2a1520a78558757a97962127ff94218bb1bb63666232c07023b99af"}
Sep 18 16:30:00.522426 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:00.522352    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95" podStartSLOduration=2.172856276 podStartE2EDuration="17.52231056s" totalImagesPullingTime="15.349454284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:29:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:00.522144474 +0000 UTC m=+834.297576655" watchObservedRunningTime="2026-09-18 16:30:00.52231056 +0000 UTC m=+834.297742722"
Sep 18 16:30:01.509498 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:01.509461    2562 generic.go:356] "Generic (PLEG): container finished" podID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerID="2bb79f06c2a1520a78558757a97962127ff94218bb1bb63666232c07023b99af" exitCode=0
Sep 18 16:30:01.509894 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:01.509507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95" event={"ID":"b87c925c-c5f6-4b70-97c9-b795c672c0cc","Type":"ContainerDied","Data":"2bb79f06c2a1520a78558757a97962127ff94218bb1bb63666232c07023b99af"}
Sep 18 16:30:02.640848 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.640816    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:30:02.800511 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.800417    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db799\" (UniqueName: \"kubernetes.io/projected/b87c925c-c5f6-4b70-97c9-b795c672c0cc-kube-api-access-db799\") pod \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") "
Sep 18 16:30:02.800511 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.800510    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-util\") pod \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") "
Sep 18 16:30:02.800698 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.800547    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-bundle\") pod \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\" (UID: \"b87c925c-c5f6-4b70-97c9-b795c672c0cc\") "
Sep 18 16:30:02.801205 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.801178    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-bundle" (OuterVolumeSpecName: "bundle") pod "b87c925c-c5f6-4b70-97c9-b795c672c0cc" (UID: "b87c925c-c5f6-4b70-97c9-b795c672c0cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:02.802716 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.802674    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87c925c-c5f6-4b70-97c9-b795c672c0cc-kube-api-access-db799" (OuterVolumeSpecName: "kube-api-access-db799") pod "b87c925c-c5f6-4b70-97c9-b795c672c0cc" (UID: "b87c925c-c5f6-4b70-97c9-b795c672c0cc"). InnerVolumeSpecName "kube-api-access-db799". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:30:02.805185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.805153    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-util" (OuterVolumeSpecName: "util") pod "b87c925c-c5f6-4b70-97c9-b795c672c0cc" (UID: "b87c925c-c5f6-4b70-97c9-b795c672c0cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:30:02.901289 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.901214    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-util\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:30:02.901289 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.901283    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b87c925c-c5f6-4b70-97c9-b795c672c0cc-bundle\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:30:02.901289 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:02.901298    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db799\" (UniqueName: \"kubernetes.io/projected/b87c925c-c5f6-4b70-97c9-b795c672c0cc-kube-api-access-db799\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:30:03.518942 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:03.518915    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95"
Sep 18 16:30:03.519120 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:03.518903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a8sx95" event={"ID":"b87c925c-c5f6-4b70-97c9-b795c672c0cc","Type":"ContainerDied","Data":"d8cb3dfeb2908b55a53ed753604158ac1f6ff77e977bef2c115f34578bb984ba"}
Sep 18 16:30:03.519120 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:03.519038    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8cb3dfeb2908b55a53ed753604158ac1f6ff77e977bef2c115f34578bb984ba"
Sep 18 16:30:06.576515 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576476    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"]
Sep 18 16:30:06.576902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576796    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerName="pull"
Sep 18 16:30:06.576902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576810    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerName="pull"
Sep 18 16:30:06.576902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576821    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerName="extract"
Sep 18 16:30:06.576902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576827    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerName="extract"
Sep 18 16:30:06.576902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576838    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerName="util"
Sep 18 16:30:06.576902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576843    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerName="util"
Sep 18 16:30:06.576902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.576888    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87c925c-c5f6-4b70-97c9-b795c672c0cc" containerName="extract"
Sep 18 16:30:06.617524 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.617480    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"]
Sep 18 16:30:06.617700 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.617499    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:06.620654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.620626    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 18 16:30:06.620797 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.620634    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 18 16:30:06.620972 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.620955    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 18 16:30:06.621106 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.621091    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-qkrq4\""
Sep 18 16:30:06.729692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.729645    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b964fc4b-a140-4bba-91c7-229aa2b5f5ee-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jv5l4\" (UID: \"b964fc4b-a140-4bba-91c7-229aa2b5f5ee\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:06.729692 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.729693    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cbkt\" (UniqueName: \"kubernetes.io/projected/b964fc4b-a140-4bba-91c7-229aa2b5f5ee-kube-api-access-7cbkt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jv5l4\" (UID: \"b964fc4b-a140-4bba-91c7-229aa2b5f5ee\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:06.830382 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.830292    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b964fc4b-a140-4bba-91c7-229aa2b5f5ee-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jv5l4\" (UID: \"b964fc4b-a140-4bba-91c7-229aa2b5f5ee\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:06.830382 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.830330    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cbkt\" (UniqueName: \"kubernetes.io/projected/b964fc4b-a140-4bba-91c7-229aa2b5f5ee-kube-api-access-7cbkt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jv5l4\" (UID: \"b964fc4b-a140-4bba-91c7-229aa2b5f5ee\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:06.832734 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.832714    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b964fc4b-a140-4bba-91c7-229aa2b5f5ee-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jv5l4\" (UID: \"b964fc4b-a140-4bba-91c7-229aa2b5f5ee\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:06.876199 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.876159    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cbkt\" (UniqueName: \"kubernetes.io/projected/b964fc4b-a140-4bba-91c7-229aa2b5f5ee-kube-api-access-7cbkt\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-jv5l4\" (UID: \"b964fc4b-a140-4bba-91c7-229aa2b5f5ee\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:06.928891 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:06.928853    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:07.237911 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:07.237880    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"]
Sep 18 16:30:07.240789 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:30:07.240759    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb964fc4b_a140_4bba_91c7_229aa2b5f5ee.slice/crio-d42d9462b07885ea154b4e93965280548dd593a4be0a2d9ed43b220c16f29bd0 WatchSource:0}: Error finding container d42d9462b07885ea154b4e93965280548dd593a4be0a2d9ed43b220c16f29bd0: Status 404 returned error can't find the container with id d42d9462b07885ea154b4e93965280548dd593a4be0a2d9ed43b220c16f29bd0
Sep 18 16:30:07.536417 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:07.536327    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4" event={"ID":"b964fc4b-a140-4bba-91c7-229aa2b5f5ee","Type":"ContainerStarted","Data":"d42d9462b07885ea154b4e93965280548dd593a4be0a2d9ed43b220c16f29bd0"}
Sep 18 16:30:11.561990 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.561946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4" event={"ID":"b964fc4b-a140-4bba-91c7-229aa2b5f5ee","Type":"ContainerStarted","Data":"32592da4c95c69ab116da017602f370f4c4269b82b72f86aada29759c845ce96"}
Sep 18 16:30:11.562439 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.562054    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:11.602627 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.602568    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4" podStartSLOduration=1.9978228630000001 podStartE2EDuration="5.602550785s" totalImagesPullingTime="3.604727922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:11.60125448 +0000 UTC m=+845.376686657" watchObservedRunningTime="2026-09-18 16:30:11.602550785 +0000 UTC m=+845.377982946"
Sep 18 16:30:11.722098 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.722053    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2k5nh"]
Sep 18 16:30:11.724613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.724586    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.727158 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.727129    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vdlct\""
Sep 18 16:30:11.727319 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.727129    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 18 16:30:11.727319 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.727177    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 18 16:30:11.736702 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.736668    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2k5nh"]
Sep 18 16:30:11.768602 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.768556    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bj2m\" (UniqueName: \"kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-kube-api-access-8bj2m\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.768799 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.768617    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-certificates\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.768799 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.768659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/df4c74ef-3cb5-4664-b522-66ff674275bd-cabundle0\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.869246 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.869213    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bj2m\" (UniqueName: \"kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-kube-api-access-8bj2m\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.869439 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.869285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-certificates\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.869439 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.869305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/df4c74ef-3cb5-4664-b522-66ff674275bd-cabundle0\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.869439 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:11.869423    2562 secret.go:281] references non-existent secret key: ca.crt
Sep 18 16:30:11.869599 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:11.869446    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 18 16:30:11.869599 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:11.869460    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-2k5nh: references non-existent secret key: ca.crt
Sep 18 16:30:11.869599 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:11.869512    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-certificates podName:df4c74ef-3cb5-4664-b522-66ff674275bd nodeName:}" failed. No retries permitted until 2026-09-18 16:30:12.369495184 +0000 UTC m=+846.144927324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-certificates") pod "keda-operator-579c8958f7-2k5nh" (UID: "df4c74ef-3cb5-4664-b522-66ff674275bd") : references non-existent secret key: ca.crt
Sep 18 16:30:11.869916 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.869897    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/df4c74ef-3cb5-4664-b522-66ff674275bd-cabundle0\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:11.881677 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:11.881644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bj2m\" (UniqueName: \"kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-kube-api-access-8bj2m\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:12.030822 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.030749    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"]
Sep 18 16:30:12.033306 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.033280    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.036439 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.036410    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 18 16:30:12.044411 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.044364    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"]
Sep 18 16:30:12.070779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.070729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/166a8181-dca7-4a66-848d-29a078711148-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.070963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.070790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6sj2\" (UniqueName: \"kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-kube-api-access-f6sj2\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.070963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.070857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.171399 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.171306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/166a8181-dca7-4a66-848d-29a078711148-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.171399 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.171364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6sj2\" (UniqueName: \"kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-kube-api-access-f6sj2\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.171613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.171401    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.171613 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:12.171536    2562 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Sep 18 16:30:12.171613 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:12.171562    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp: secret "keda-metrics-apiserver-certs" not found
Sep 18 16:30:12.171613 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:12.171614    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-certificates podName:166a8181-dca7-4a66-848d-29a078711148 nodeName:}" failed. No retries permitted until 2026-09-18 16:30:12.67159661 +0000 UTC m=+846.447028750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-certificates") pod "keda-metrics-apiserver-77857b7b98-kdmtp" (UID: "166a8181-dca7-4a66-848d-29a078711148") : secret "keda-metrics-apiserver-certs" not found
Sep 18 16:30:12.171803 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.171699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/166a8181-dca7-4a66-848d-29a078711148-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.181548 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.181512    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6sj2\" (UniqueName: \"kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-kube-api-access-f6sj2\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.373473 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.373434    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-certificates\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:12.376213 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.376185    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/df4c74ef-3cb5-4664-b522-66ff674275bd-certificates\") pod \"keda-operator-579c8958f7-2k5nh\" (UID: \"df4c74ef-3cb5-4664-b522-66ff674275bd\") " pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:12.410048 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.410015    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jxf8d"]
Sep 18 16:30:12.412503 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.412479    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:12.415221 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.415194    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 18 16:30:12.424799 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.424727    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jxf8d"]
Sep 18 16:30:12.474876 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.474836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghpx\" (UniqueName: \"kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-kube-api-access-bghpx\") pod \"keda-admission-6c4d864b86-jxf8d\" (UID: \"fce08193-e9bb-486a-91a1-f75aa20620fb\") " pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:12.475055 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.474884    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-certificates\") pod \"keda-admission-6c4d864b86-jxf8d\" (UID: \"fce08193-e9bb-486a-91a1-f75aa20620fb\") " pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:12.575738 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.575701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghpx\" (UniqueName: \"kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-kube-api-access-bghpx\") pod \"keda-admission-6c4d864b86-jxf8d\" (UID: \"fce08193-e9bb-486a-91a1-f75aa20620fb\") " pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:12.576225 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.575747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-certificates\") pod \"keda-admission-6c4d864b86-jxf8d\" (UID: \"fce08193-e9bb-486a-91a1-f75aa20620fb\") " pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:12.576225 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:12.575910    2562 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 18 16:30:12.576225 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:12.575942    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-jxf8d: secret "keda-admission-webhooks-certs" not found
Sep 18 16:30:12.576225 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:30:12.576000    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-certificates podName:fce08193-e9bb-486a-91a1-f75aa20620fb nodeName:}" failed. No retries permitted until 2026-09-18 16:30:13.075982682 +0000 UTC m=+846.851414839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-certificates") pod "keda-admission-6c4d864b86-jxf8d" (UID: "fce08193-e9bb-486a-91a1-f75aa20620fb") : secret "keda-admission-webhooks-certs" not found
Sep 18 16:30:12.593347 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.593319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghpx\" (UniqueName: \"kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-kube-api-access-bghpx\") pod \"keda-admission-6c4d864b86-jxf8d\" (UID: \"fce08193-e9bb-486a-91a1-f75aa20620fb\") " pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:12.634966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.634926    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:12.676675 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.676588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.679262 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.679205    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/166a8181-dca7-4a66-848d-29a078711148-certificates\") pod \"keda-metrics-apiserver-77857b7b98-kdmtp\" (UID: \"166a8181-dca7-4a66-848d-29a078711148\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:12.928252 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.928210    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-2k5nh"]
Sep 18 16:30:12.931218 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:30:12.931193    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4c74ef_3cb5_4664_b522_66ff674275bd.slice/crio-51dbbbdc0cee0af3489336a88e3c32d6ab5d5f0d30892dc2fed9c73d85de5156 WatchSource:0}: Error finding container 51dbbbdc0cee0af3489336a88e3c32d6ab5d5f0d30892dc2fed9c73d85de5156: Status 404 returned error can't find the container with id 51dbbbdc0cee0af3489336a88e3c32d6ab5d5f0d30892dc2fed9c73d85de5156
Sep 18 16:30:12.946093 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:12.946063    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:13.079918 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:13.079879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-certificates\") pod \"keda-admission-6c4d864b86-jxf8d\" (UID: \"fce08193-e9bb-486a-91a1-f75aa20620fb\") " pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:13.082564 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:13.082535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fce08193-e9bb-486a-91a1-f75aa20620fb-certificates\") pod \"keda-admission-6c4d864b86-jxf8d\" (UID: \"fce08193-e9bb-486a-91a1-f75aa20620fb\") " pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:13.225327 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:13.225221    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"]
Sep 18 16:30:13.228606 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:30:13.228560    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166a8181_dca7_4a66_848d_29a078711148.slice/crio-31ff9a7b3d48d841ec5ed92edcc5e037ea030fd99220a66858afd9804396b2d9 WatchSource:0}: Error finding container 31ff9a7b3d48d841ec5ed92edcc5e037ea030fd99220a66858afd9804396b2d9: Status 404 returned error can't find the container with id 31ff9a7b3d48d841ec5ed92edcc5e037ea030fd99220a66858afd9804396b2d9
Sep 18 16:30:13.324169 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:13.324131    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:13.571530 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:13.571427    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp" event={"ID":"166a8181-dca7-4a66-848d-29a078711148","Type":"ContainerStarted","Data":"31ff9a7b3d48d841ec5ed92edcc5e037ea030fd99220a66858afd9804396b2d9"}
Sep 18 16:30:13.572673 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:13.572644    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2k5nh" event={"ID":"df4c74ef-3cb5-4664-b522-66ff674275bd","Type":"ContainerStarted","Data":"51dbbbdc0cee0af3489336a88e3c32d6ab5d5f0d30892dc2fed9c73d85de5156"}
Sep 18 16:30:13.604471 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:13.604443    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-jxf8d"]
Sep 18 16:30:13.607116 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:30:13.607082    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce08193_e9bb_486a_91a1_f75aa20620fb.slice/crio-98b60057e615c9fa9a1ac40193a2894ccbec0e68356925729a29d25209f2785b WatchSource:0}: Error finding container 98b60057e615c9fa9a1ac40193a2894ccbec0e68356925729a29d25209f2785b: Status 404 returned error can't find the container with id 98b60057e615c9fa9a1ac40193a2894ccbec0e68356925729a29d25209f2785b
Sep 18 16:30:14.580148 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:14.580109    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jxf8d" event={"ID":"fce08193-e9bb-486a-91a1-f75aa20620fb","Type":"ContainerStarted","Data":"98b60057e615c9fa9a1ac40193a2894ccbec0e68356925729a29d25209f2785b"}
Sep 18 16:30:16.592287 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:16.592224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp" event={"ID":"166a8181-dca7-4a66-848d-29a078711148","Type":"ContainerStarted","Data":"bb037ff4561eef949c33575ee51073fa20ed7dc6b260a414fe727910f96b497b"}
Sep 18 16:30:16.592705 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:16.592370    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:16.613655 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:16.613598    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp" podStartSLOduration=2.030078738 podStartE2EDuration="4.613579987s" totalImagesPullingTime="2.583501249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:16.612029116 +0000 UTC m=+850.387461290" watchObservedRunningTime="2026-09-18 16:30:16.613579987 +0000 UTC m=+850.389012148"
Sep 18 16:30:24.636599 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:24.636549    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-jxf8d" event={"ID":"fce08193-e9bb-486a-91a1-f75aa20620fb","Type":"ContainerStarted","Data":"aa064e499c8e12743d7254b14c7f85b18c2cfa070fe5ef092765817f0b145214"}
Sep 18 16:30:24.637089 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:24.636716    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:24.672214 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:24.672149    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-jxf8d" podStartSLOduration=2.643628557 podStartE2EDuration="12.672124401s" totalImagesPullingTime="10.028495844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:24.670960307 +0000 UTC m=+858.446392497" watchObservedRunningTime="2026-09-18 16:30:24.672124401 +0000 UTC m=+858.447556565"
Sep 18 16:30:26.648473 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:26.648426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-2k5nh" event={"ID":"df4c74ef-3cb5-4664-b522-66ff674275bd","Type":"ContainerStarted","Data":"4131482ad6b4346360230b8dd322d34f37b36f79b6ca52f671085614918c68de"}
Sep 18 16:30:26.648873 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:26.648540    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:30:26.673504 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:26.673390    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-2k5nh" podStartSLOduration=2.187827775 podStartE2EDuration="15.673360327s" totalImagesPullingTime="13.485532552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:30:26.671477613 +0000 UTC m=+860.446909796" watchObservedRunningTime="2026-09-18 16:30:26.673360327 +0000 UTC m=+860.448792490"
Sep 18 16:30:27.603048 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:27.603020    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-kdmtp"
Sep 18 16:30:32.569395 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:32.569363    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-jv5l4"
Sep 18 16:30:45.645063 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:45.645028    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-jxf8d"
Sep 18 16:30:47.656055 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:30:47.656022    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-2k5nh"
Sep 18 16:31:03.405445 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.405404    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"]
Sep 18 16:31:03.410637 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.410613    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.413968 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.413935    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 16:31:03.414185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.414171    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 16:31:03.415628 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.415607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jv47d\""
Sep 18 16:31:03.419988 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.419963    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"]
Sep 18 16:31:03.513384 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.513344    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.513609 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.513400    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.513609 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.513453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52tmv\" (UniqueName: \"kubernetes.io/projected/3ff76e27-130f-41d0-8f61-02e206fdac4e-kube-api-access-52tmv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.614346 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.614313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.614461 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.614358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.614501 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.614474    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52tmv\" (UniqueName: \"kubernetes.io/projected/3ff76e27-130f-41d0-8f61-02e206fdac4e-kube-api-access-52tmv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.614714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.614695    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.614751 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.614729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.624226 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.624194    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52tmv\" (UniqueName: \"kubernetes.io/projected/3ff76e27-130f-41d0-8f61-02e206fdac4e-kube-api-access-52tmv\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:03.720661 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:03.720570    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:04.012299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:04.012267    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"]
Sep 18 16:31:04.014718 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:31:04.014692    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff76e27_130f_41d0_8f61_02e206fdac4e.slice/crio-e20859788691be21d34d7a5d433486b8a61527fe426c3884bf33e3628cf0cf40 WatchSource:0}: Error finding container e20859788691be21d34d7a5d433486b8a61527fe426c3884bf33e3628cf0cf40: Status 404 returned error can't find the container with id e20859788691be21d34d7a5d433486b8a61527fe426c3884bf33e3628cf0cf40
Sep 18 16:31:04.823481 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:04.823446    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerID="867388c35e70bfbc00456e4808b8918e118a9bd7801bac1565ef09f9b0dbe439" exitCode=0
Sep 18 16:31:04.823889 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:04.823537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv" event={"ID":"3ff76e27-130f-41d0-8f61-02e206fdac4e","Type":"ContainerDied","Data":"867388c35e70bfbc00456e4808b8918e118a9bd7801bac1565ef09f9b0dbe439"}
Sep 18 16:31:04.823889 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:04.823596    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv" event={"ID":"3ff76e27-130f-41d0-8f61-02e206fdac4e","Type":"ContainerStarted","Data":"e20859788691be21d34d7a5d433486b8a61527fe426c3884bf33e3628cf0cf40"}
Sep 18 16:31:04.824706 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:04.824688    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:31:07.845130 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:07.845094    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerID="d483c78dd9978b1c556eecacdcf3c050531679aec5e90f6be7a354e8a9f1c868" exitCode=0
Sep 18 16:31:07.845543 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:07.845173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv" event={"ID":"3ff76e27-130f-41d0-8f61-02e206fdac4e","Type":"ContainerDied","Data":"d483c78dd9978b1c556eecacdcf3c050531679aec5e90f6be7a354e8a9f1c868"}
Sep 18 16:31:08.857425 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:08.857386    2562 generic.go:356] "Generic (PLEG): container finished" podID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerID="955850ce3cbb881c42e02af44398b34e60e25bdfce8129810d04e5d25beab6d3" exitCode=0
Sep 18 16:31:08.857807 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:08.857463    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv" event={"ID":"3ff76e27-130f-41d0-8f61-02e206fdac4e","Type":"ContainerDied","Data":"955850ce3cbb881c42e02af44398b34e60e25bdfce8129810d04e5d25beab6d3"}
Sep 18 16:31:09.993338 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:09.993301    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:10.166117 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.166081    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52tmv\" (UniqueName: \"kubernetes.io/projected/3ff76e27-130f-41d0-8f61-02e206fdac4e-kube-api-access-52tmv\") pod \"3ff76e27-130f-41d0-8f61-02e206fdac4e\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") "
Sep 18 16:31:10.166356 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.166171    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-util\") pod \"3ff76e27-130f-41d0-8f61-02e206fdac4e\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") "
Sep 18 16:31:10.166356 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.166201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-bundle\") pod \"3ff76e27-130f-41d0-8f61-02e206fdac4e\" (UID: \"3ff76e27-130f-41d0-8f61-02e206fdac4e\") "
Sep 18 16:31:10.166939 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.166915    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-bundle" (OuterVolumeSpecName: "bundle") pod "3ff76e27-130f-41d0-8f61-02e206fdac4e" (UID: "3ff76e27-130f-41d0-8f61-02e206fdac4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:31:10.168398 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.168361    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff76e27-130f-41d0-8f61-02e206fdac4e-kube-api-access-52tmv" (OuterVolumeSpecName: "kube-api-access-52tmv") pod "3ff76e27-130f-41d0-8f61-02e206fdac4e" (UID: "3ff76e27-130f-41d0-8f61-02e206fdac4e"). InnerVolumeSpecName "kube-api-access-52tmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:31:10.170729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.170688    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-util" (OuterVolumeSpecName: "util") pod "3ff76e27-130f-41d0-8f61-02e206fdac4e" (UID: "3ff76e27-130f-41d0-8f61-02e206fdac4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:31:10.267687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.267655    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-bundle\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:31:10.267687 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.267687    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52tmv\" (UniqueName: \"kubernetes.io/projected/3ff76e27-130f-41d0-8f61-02e206fdac4e-kube-api-access-52tmv\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:31:10.267907 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.267699    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ff76e27-130f-41d0-8f61-02e206fdac4e-util\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:31:10.867579 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.867544    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv" event={"ID":"3ff76e27-130f-41d0-8f61-02e206fdac4e","Type":"ContainerDied","Data":"e20859788691be21d34d7a5d433486b8a61527fe426c3884bf33e3628cf0cf40"}
Sep 18 16:31:10.867579 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.867573    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ptktv"
Sep 18 16:31:10.867807 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:10.867594    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e20859788691be21d34d7a5d433486b8a61527fe426c3884bf33e3628cf0cf40"
Sep 18 16:31:16.556891 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.556853    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"]
Sep 18 16:31:16.557372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.557175    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerName="util"
Sep 18 16:31:16.557372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.557185    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerName="util"
Sep 18 16:31:16.557372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.557194    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerName="pull"
Sep 18 16:31:16.557372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.557200    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerName="pull"
Sep 18 16:31:16.557372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.557219    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerName="extract"
Sep 18 16:31:16.557372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.557225    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerName="extract"
Sep 18 16:31:16.557372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.557288    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff76e27-130f-41d0-8f61-02e206fdac4e" containerName="extract"
Sep 18 16:31:16.559957 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.559939    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:16.562896 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.562873    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:31:16.563028 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.562973    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 18 16:31:16.563195 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.563180    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-gbd85\""
Sep 18 16:31:16.575393 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.575361    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"]
Sep 18 16:31:16.617718 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.617666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlhb6\" (UniqueName: \"kubernetes.io/projected/920d7c8d-2116-4c2a-acb9-6e33a664290a-kube-api-access-wlhb6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5ptm6\" (UID: \"920d7c8d-2116-4c2a-acb9-6e33a664290a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:16.617907 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.617731    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/920d7c8d-2116-4c2a-acb9-6e33a664290a-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5ptm6\" (UID: \"920d7c8d-2116-4c2a-acb9-6e33a664290a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:16.718295 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.718257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlhb6\" (UniqueName: \"kubernetes.io/projected/920d7c8d-2116-4c2a-acb9-6e33a664290a-kube-api-access-wlhb6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5ptm6\" (UID: \"920d7c8d-2116-4c2a-acb9-6e33a664290a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:16.718448 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.718326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/920d7c8d-2116-4c2a-acb9-6e33a664290a-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5ptm6\" (UID: \"920d7c8d-2116-4c2a-acb9-6e33a664290a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:16.718742 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.718719    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/920d7c8d-2116-4c2a-acb9-6e33a664290a-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5ptm6\" (UID: \"920d7c8d-2116-4c2a-acb9-6e33a664290a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:16.744801 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.744765    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlhb6\" (UniqueName: \"kubernetes.io/projected/920d7c8d-2116-4c2a-acb9-6e33a664290a-kube-api-access-wlhb6\") pod \"cert-manager-operator-controller-manager-684f79c9b9-5ptm6\" (UID: \"920d7c8d-2116-4c2a-acb9-6e33a664290a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:16.869902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:16.869868    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"
Sep 18 16:31:17.183644 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:17.183616    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6"]
Sep 18 16:31:17.185262 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:31:17.185217    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920d7c8d_2116_4c2a_acb9_6e33a664290a.slice/crio-b990c9e1dda14d4bcdd6818eae85b90fec346588e10dbc87decb39c3da851cf2 WatchSource:0}: Error finding container b990c9e1dda14d4bcdd6818eae85b90fec346588e10dbc87decb39c3da851cf2: Status 404 returned error can't find the container with id b990c9e1dda14d4bcdd6818eae85b90fec346588e10dbc87decb39c3da851cf2
Sep 18 16:31:17.901672 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:17.901635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6" event={"ID":"920d7c8d-2116-4c2a-acb9-6e33a664290a","Type":"ContainerStarted","Data":"b990c9e1dda14d4bcdd6818eae85b90fec346588e10dbc87decb39c3da851cf2"}
Sep 18 16:31:18.909473 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:18.909440    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6" event={"ID":"920d7c8d-2116-4c2a-acb9-6e33a664290a","Type":"ContainerStarted","Data":"e0e807ed61fbd36f02147e65751302ae5edaefb14f47eb7595e60c255bcb83cf"}
Sep 18 16:31:18.936580 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:18.936524    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-5ptm6" podStartSLOduration=1.335377975 podStartE2EDuration="2.936507716s" totalImagesPullingTime="1.601129741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:18.933981393 +0000 UTC m=+912.709413616" watchObservedRunningTime="2026-09-18 16:31:18.936507716 +0000 UTC m=+912.711939872"
Sep 18 16:31:25.828532 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.828488    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"]
Sep 18 16:31:25.831205 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.831181    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.834029 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.834007    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 16:31:25.835064 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.835047    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jv47d\""
Sep 18 16:31:25.835124 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.835048    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 16:31:25.841498 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.841471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"]
Sep 18 16:31:25.889286 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.889219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.889466 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.889315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.889466 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.889349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6dx\" (UniqueName: \"kubernetes.io/projected/adacfef6-5017-4882-ae9b-ce1da5ad5776-kube-api-access-vc6dx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.990407 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.990374    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6dx\" (UniqueName: \"kubernetes.io/projected/adacfef6-5017-4882-ae9b-ce1da5ad5776-kube-api-access-vc6dx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.990407 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.990420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.990616 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.990468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.990838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.990822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:25.990886 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:25.990867    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:26.000514 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:26.000476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6dx\" (UniqueName: \"kubernetes.io/projected/adacfef6-5017-4882-ae9b-ce1da5ad5776-kube-api-access-vc6dx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:26.140387 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:26.140352    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:26.433699 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:26.433673    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"]
Sep 18 16:31:26.436357 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:31:26.436321    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadacfef6_5017_4882_ae9b_ce1da5ad5776.slice/crio-befe5e1e0b50f80daeaeb86516c35c3ed741c2101521f5cd4b9fd58a4fc305f7 WatchSource:0}: Error finding container befe5e1e0b50f80daeaeb86516c35c3ed741c2101521f5cd4b9fd58a4fc305f7: Status 404 returned error can't find the container with id befe5e1e0b50f80daeaeb86516c35c3ed741c2101521f5cd4b9fd58a4fc305f7
Sep 18 16:31:26.946747 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:26.946711    2562 generic.go:356] "Generic (PLEG): container finished" podID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerID="f094a188ac4d6991924231b5d9067bcd4d800cf9bf3d82633efdd5bfbcc7e940" exitCode=0
Sep 18 16:31:26.947124 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:26.946793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp" event={"ID":"adacfef6-5017-4882-ae9b-ce1da5ad5776","Type":"ContainerDied","Data":"f094a188ac4d6991924231b5d9067bcd4d800cf9bf3d82633efdd5bfbcc7e940"}
Sep 18 16:31:26.947124 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:26.946831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp" event={"ID":"adacfef6-5017-4882-ae9b-ce1da5ad5776","Type":"ContainerStarted","Data":"befe5e1e0b50f80daeaeb86516c35c3ed741c2101521f5cd4b9fd58a4fc305f7"}
Sep 18 16:31:29.963348 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:29.963308    2562 generic.go:356] "Generic (PLEG): container finished" podID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerID="dc134f6f426f4c433eb27d4cde9b9e65c1ed3ffed98ac6fd07a4e5ec14f063a5" exitCode=0
Sep 18 16:31:29.963742 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:29.963398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp" event={"ID":"adacfef6-5017-4882-ae9b-ce1da5ad5776","Type":"ContainerDied","Data":"dc134f6f426f4c433eb27d4cde9b9e65c1ed3ffed98ac6fd07a4e5ec14f063a5"}
Sep 18 16:31:30.970071 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:30.970035    2562 generic.go:356] "Generic (PLEG): container finished" podID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerID="9a81e0a89c622ee97027ef73a5b6027d6a14ad86fc5db28b01669c3582844219" exitCode=0
Sep 18 16:31:30.970469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:30.970120    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp" event={"ID":"adacfef6-5017-4882-ae9b-ce1da5ad5776","Type":"ContainerDied","Data":"9a81e0a89c622ee97027ef73a5b6027d6a14ad86fc5db28b01669c3582844219"}
Sep 18 16:31:32.105039 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.105014    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:32.146438 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.146406    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-util\") pod \"adacfef6-5017-4882-ae9b-ce1da5ad5776\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") "
Sep 18 16:31:32.146619 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.146465    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-bundle\") pod \"adacfef6-5017-4882-ae9b-ce1da5ad5776\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") "
Sep 18 16:31:32.146619 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.146530    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vc6dx\" (UniqueName: \"kubernetes.io/projected/adacfef6-5017-4882-ae9b-ce1da5ad5776-kube-api-access-vc6dx\") pod \"adacfef6-5017-4882-ae9b-ce1da5ad5776\" (UID: \"adacfef6-5017-4882-ae9b-ce1da5ad5776\") "
Sep 18 16:31:32.146964 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.146932    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-bundle" (OuterVolumeSpecName: "bundle") pod "adacfef6-5017-4882-ae9b-ce1da5ad5776" (UID: "adacfef6-5017-4882-ae9b-ce1da5ad5776"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:31:32.148749 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.148713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adacfef6-5017-4882-ae9b-ce1da5ad5776-kube-api-access-vc6dx" (OuterVolumeSpecName: "kube-api-access-vc6dx") pod "adacfef6-5017-4882-ae9b-ce1da5ad5776" (UID: "adacfef6-5017-4882-ae9b-ce1da5ad5776"). InnerVolumeSpecName "kube-api-access-vc6dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:31:32.151273 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.151225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-util" (OuterVolumeSpecName: "util") pod "adacfef6-5017-4882-ae9b-ce1da5ad5776" (UID: "adacfef6-5017-4882-ae9b-ce1da5ad5776"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:31:32.248108 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.248010    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-util\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:31:32.248108 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.248046    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/adacfef6-5017-4882-ae9b-ce1da5ad5776-bundle\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:31:32.248108 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.248055    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc6dx\" (UniqueName: \"kubernetes.io/projected/adacfef6-5017-4882-ae9b-ce1da5ad5776-kube-api-access-vc6dx\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:31:32.983099 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.983065    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp"
Sep 18 16:31:32.983309 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.983065    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fv4jrp" event={"ID":"adacfef6-5017-4882-ae9b-ce1da5ad5776","Type":"ContainerDied","Data":"befe5e1e0b50f80daeaeb86516c35c3ed741c2101521f5cd4b9fd58a4fc305f7"}
Sep 18 16:31:32.983309 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:32.983192    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="befe5e1e0b50f80daeaeb86516c35c3ed741c2101521f5cd4b9fd58a4fc305f7"
Sep 18 16:31:37.218229 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218187    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"]
Sep 18 16:31:37.218785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218579    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerName="pull"
Sep 18 16:31:37.218785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218596    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerName="pull"
Sep 18 16:31:37.218785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218606    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerName="util"
Sep 18 16:31:37.218785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218615    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerName="util"
Sep 18 16:31:37.218785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218633    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerName="extract"
Sep 18 16:31:37.218785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218640    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerName="extract"
Sep 18 16:31:37.218785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.218693    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="adacfef6-5017-4882-ae9b-ce1da5ad5776" containerName="extract"
Sep 18 16:31:37.222045 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.222026    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.224915 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.224884    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 16:31:37.225051 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.224983    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 16:31:37.225051 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.225000    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fkr6h\""
Sep 18 16:31:37.232194 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.232165    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"]
Sep 18 16:31:37.291896 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.291845    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm2nd\" (UniqueName: \"kubernetes.io/projected/c5b6d717-261d-4ef2-9f4c-b0addfa22d97-kube-api-access-qm2nd\") pod \"openshift-lws-operator-fd8ccff4c-g22cc\" (UID: \"c5b6d717-261d-4ef2-9f4c-b0addfa22d97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.292072 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.291954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5b6d717-261d-4ef2-9f4c-b0addfa22d97-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g22cc\" (UID: \"c5b6d717-261d-4ef2-9f4c-b0addfa22d97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.393199 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.393159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5b6d717-261d-4ef2-9f4c-b0addfa22d97-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g22cc\" (UID: \"c5b6d717-261d-4ef2-9f4c-b0addfa22d97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.393389 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.393226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2nd\" (UniqueName: \"kubernetes.io/projected/c5b6d717-261d-4ef2-9f4c-b0addfa22d97-kube-api-access-qm2nd\") pod \"openshift-lws-operator-fd8ccff4c-g22cc\" (UID: \"c5b6d717-261d-4ef2-9f4c-b0addfa22d97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.393557 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.393531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5b6d717-261d-4ef2-9f4c-b0addfa22d97-tmp\") pod \"openshift-lws-operator-fd8ccff4c-g22cc\" (UID: \"c5b6d717-261d-4ef2-9f4c-b0addfa22d97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.402164 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.402134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2nd\" (UniqueName: \"kubernetes.io/projected/c5b6d717-261d-4ef2-9f4c-b0addfa22d97-kube-api-access-qm2nd\") pod \"openshift-lws-operator-fd8ccff4c-g22cc\" (UID: \"c5b6d717-261d-4ef2-9f4c-b0addfa22d97\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.545641 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.545556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"
Sep 18 16:31:37.818698 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:37.818673    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc"]
Sep 18 16:31:37.821313 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:31:37.821277    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5b6d717_261d_4ef2_9f4c_b0addfa22d97.slice/crio-dd81bd6c8b2f562484ccd6c7324cf07cc68613c28a41bb2190fd57cfa718903f WatchSource:0}: Error finding container dd81bd6c8b2f562484ccd6c7324cf07cc68613c28a41bb2190fd57cfa718903f: Status 404 returned error can't find the container with id dd81bd6c8b2f562484ccd6c7324cf07cc68613c28a41bb2190fd57cfa718903f
Sep 18 16:31:38.007982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:38.007944    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc" event={"ID":"c5b6d717-261d-4ef2-9f4c-b0addfa22d97","Type":"ContainerStarted","Data":"dd81bd6c8b2f562484ccd6c7324cf07cc68613c28a41bb2190fd57cfa718903f"}
Sep 18 16:31:40.019761 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:40.019716    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc" event={"ID":"c5b6d717-261d-4ef2-9f4c-b0addfa22d97","Type":"ContainerStarted","Data":"b5a6f5dfaecba78009f89415cc0e20f4221e9be1f8bca2b8611aea3869b54a73"}
Sep 18 16:31:40.043419 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:40.043349    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-g22cc" podStartSLOduration=1.409719438 podStartE2EDuration="3.043325814s" totalImagesPullingTime="1.633606376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:40.043186349 +0000 UTC m=+933.818618515" watchObservedRunningTime="2026-09-18 16:31:40.043325814 +0000 UTC m=+933.818757998"
Sep 18 16:31:47.770060 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.770023    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"]
Sep 18 16:31:47.776199 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.776173    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.779086 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.779058    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 18 16:31:47.779256 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.779059    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 18 16:31:47.779256 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.779059    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 18 16:31:47.779256 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.779071    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 16:31:47.779524 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.779507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-v97lk\""
Sep 18 16:31:47.787831 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.787803    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"]
Sep 18 16:31:47.876288 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.876212    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ggsv\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-kube-api-access-7ggsv\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.876461 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.876303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.876461 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.876326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb48228-6af3-420e-b596-620dab4086c8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.876461 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.876353    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.876461 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.876409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.876597 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.876467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.876597 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.876511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.977609 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.977567    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.977749 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.977616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb48228-6af3-420e-b596-620dab4086c8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.977749 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.977658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.977749 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.977693    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.977749 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.977737    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.978001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.977780    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.978001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.977828    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ggsv\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-kube-api-access-7ggsv\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.978505 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.978476    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.980177 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.980149    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb48228-6af3-420e-b596-620dab4086c8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.980355 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.980326    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.980574 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.980558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.980635 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.980616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.986181 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.986147    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:47.986311 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:47.986276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ggsv\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-kube-api-access-7ggsv\") pod \"istiod-openshift-gateway-7b7995d5cd-ggmfr\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:48.087908 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:48.087790    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:48.364006 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:48.363976    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"]
Sep 18 16:31:48.366254 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:31:48.366215    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb48228_6af3_420e_b596_620dab4086c8.slice/crio-93f5a594ee5277babf08d3d7957a5e97f5da49e6cdbc660ee7983d4d48160ef0 WatchSource:0}: Error finding container 93f5a594ee5277babf08d3d7957a5e97f5da49e6cdbc660ee7983d4d48160ef0: Status 404 returned error can't find the container with id 93f5a594ee5277babf08d3d7957a5e97f5da49e6cdbc660ee7983d4d48160ef0
Sep 18 16:31:49.066639 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:49.066600    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr" event={"ID":"9eb48228-6af3-420e-b596-620dab4086c8","Type":"ContainerStarted","Data":"93f5a594ee5277babf08d3d7957a5e97f5da49e6cdbc660ee7983d4d48160ef0"}
Sep 18 16:31:50.935699 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:50.935655    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 16:31:50.936050 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:50.935731    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 16:31:51.079162 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:51.079121    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr" event={"ID":"9eb48228-6af3-420e-b596-620dab4086c8","Type":"ContainerStarted","Data":"81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc"}
Sep 18 16:31:51.079365 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:51.079272    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:31:51.100671 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:51.100616    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr" podStartSLOduration=1.533664165 podStartE2EDuration="4.100596352s" totalImagesPullingTime="2.566932187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:31:51.099576932 +0000 UTC m=+944.875009118" watchObservedRunningTime="2026-09-18 16:31:51.100596352 +0000 UTC m=+944.876028513"
Sep 18 16:31:52.086571 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:52.086529    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-ggmfr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:31:52.086965 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:52.086587    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr" podUID="9eb48228-6af3-420e-b596-620dab4086c8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:31:55.085556 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:31:55.085519    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:32:06.593377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.593335    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"]
Sep 18 16:32:06.596631 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.596611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.601083 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.601053    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 18 16:32:06.601253 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.601127    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jv47d\""
Sep 18 16:32:06.602262 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.602213    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 18 16:32:06.612743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.612715    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"]
Sep 18 16:32:06.692872 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.692832    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"]
Sep 18 16:32:06.696516 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.696486    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.712371 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.712331    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"]
Sep 18 16:32:06.727671 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.727637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.727876 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.727701    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.727876 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.727749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bxj\" (UniqueName: \"kubernetes.io/projected/a4534922-1870-47d0-adae-0bffa332c0a2-kube-api-access-54bxj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.815342 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.815311    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"]
Sep 18 16:32:06.818866 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.818839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:06.829101 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829075    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.829260 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829117    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.829260 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.829260 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bxj\" (UniqueName: \"kubernetes.io/projected/a4534922-1870-47d0-adae-0bffa332c0a2-kube-api-access-54bxj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.829391 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829303    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.829391 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxkvc\" (UniqueName: \"kubernetes.io/projected/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-kube-api-access-vxkvc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.829486 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.829525 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.829510    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.836682 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.836649    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"]
Sep 18 16:32:06.844541 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.844470    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bxj\" (UniqueName: \"kubernetes.io/projected/a4534922-1870-47d0-adae-0bffa332c0a2-kube-api-access-54bxj\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.907216 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.907165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:06.929777 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.929742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.929777 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.929785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvc9\" (UniqueName: \"kubernetes.io/projected/fd319c98-354b-42c9-a105-bd3ec5a37a93-kube-api-access-sbvc9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:06.930084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.929840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:06.930084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.929875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.930084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.929921    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:06.930084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.929955    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxkvc\" (UniqueName: \"kubernetes.io/projected/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-kube-api-access-vxkvc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.930264 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.930228    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.930309 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.930272    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.941684 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.941651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxkvc\" (UniqueName: \"kubernetes.io/projected/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-kube-api-access-vxkvc\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:06.967157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.966935    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"]
Sep 18 16:32:06.972701 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.972672    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:06.986096 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:06.986040    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"]
Sep 18 16:32:07.005757 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.005714    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:07.030779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.030744    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:07.030960 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.030841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvc9\" (UniqueName: \"kubernetes.io/projected/fd319c98-354b-42c9-a105-bd3ec5a37a93-kube-api-access-sbvc9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:07.030960 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.030895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:07.031147 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.031126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:07.031302 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.031281    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:07.044471 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.044393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvc9\" (UniqueName: \"kubernetes.io/projected/fd319c98-354b-42c9-a105-bd3ec5a37a93-kube-api-access-sbvc9\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:07.128166 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.128126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:07.131514 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.131485    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.131590 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.131523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsrzh\" (UniqueName: \"kubernetes.io/projected/89bffbfd-fd0d-4b40-bd83-376194bf5c67-kube-api-access-gsrzh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.131653 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.131600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.201553 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.201519    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"]
Sep 18 16:32:07.205256 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:32:07.205190    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4534922_1870_47d0_adae_0bffa332c0a2.slice/crio-108a6aa96b53555729b3dbceb981796eec95917825c2d03cba7f3fc7e8adce83 WatchSource:0}: Error finding container 108a6aa96b53555729b3dbceb981796eec95917825c2d03cba7f3fc7e8adce83: Status 404 returned error can't find the container with id 108a6aa96b53555729b3dbceb981796eec95917825c2d03cba7f3fc7e8adce83
Sep 18 16:32:07.232154 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.232117    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.232376 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.232157    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsrzh\" (UniqueName: \"kubernetes.io/projected/89bffbfd-fd0d-4b40-bd83-376194bf5c67-kube-api-access-gsrzh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.232376 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.232211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.232605 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.232585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.232647 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.232634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.248154 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.248123    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsrzh\" (UniqueName: \"kubernetes.io/projected/89bffbfd-fd0d-4b40-bd83-376194bf5c67-kube-api-access-gsrzh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.288547 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.288511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:07.293100 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.293075    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"]
Sep 18 16:32:07.295122 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:32:07.295079    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf71f09b_45b4_487d_8f35_5b7d884dc6ed.slice/crio-37312e532261fb05e63c00118e076d279d13bba2427fe522b2691491165759eb WatchSource:0}: Error finding container 37312e532261fb05e63c00118e076d279d13bba2427fe522b2691491165759eb: Status 404 returned error can't find the container with id 37312e532261fb05e63c00118e076d279d13bba2427fe522b2691491165759eb
Sep 18 16:32:07.443704 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.443680    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"]
Sep 18 16:32:07.472230 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:32:07.472189    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd319c98_354b_42c9_a105_bd3ec5a37a93.slice/crio-d7e9bf2717dca737e96c6c3ce531cb47cb4005565bf05345738c9e987337a423 WatchSource:0}: Error finding container d7e9bf2717dca737e96c6c3ce531cb47cb4005565bf05345738c9e987337a423: Status 404 returned error can't find the container with id d7e9bf2717dca737e96c6c3ce531cb47cb4005565bf05345738c9e987337a423
Sep 18 16:32:07.593981 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:07.593953    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"]
Sep 18 16:32:07.650315 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:32:07.650204    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bffbfd_fd0d_4b40_bd83_376194bf5c67.slice/crio-f5d068162750213a772fd19bbd8dfa030abbd8a2ab41cf91bfb1758e389891ed WatchSource:0}: Error finding container f5d068162750213a772fd19bbd8dfa030abbd8a2ab41cf91bfb1758e389891ed: Status 404 returned error can't find the container with id f5d068162750213a772fd19bbd8dfa030abbd8a2ab41cf91bfb1758e389891ed
Sep 18 16:32:08.167946 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.167907    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerID="3710815f4d5e967be5e4e8e728bf40bb7644ca31d44c40a6775852433195c392" exitCode=0
Sep 18 16:32:08.168209 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.167946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7" event={"ID":"bf71f09b-45b4-487d-8f35-5b7d884dc6ed","Type":"ContainerDied","Data":"3710815f4d5e967be5e4e8e728bf40bb7644ca31d44c40a6775852433195c392"}
Sep 18 16:32:08.168209 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.167997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7" event={"ID":"bf71f09b-45b4-487d-8f35-5b7d884dc6ed","Type":"ContainerStarted","Data":"37312e532261fb05e63c00118e076d279d13bba2427fe522b2691491165759eb"}
Sep 18 16:32:08.169780 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.169756    2562 generic.go:356] "Generic (PLEG): container finished" podID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerID="afdef4219486a1156897679c5dd1a35fa5dbb21b144be649526b1980ddc75142" exitCode=0
Sep 18 16:32:08.169874 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.169857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld" event={"ID":"89bffbfd-fd0d-4b40-bd83-376194bf5c67","Type":"ContainerDied","Data":"afdef4219486a1156897679c5dd1a35fa5dbb21b144be649526b1980ddc75142"}
Sep 18 16:32:08.169939 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.169903    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld" event={"ID":"89bffbfd-fd0d-4b40-bd83-376194bf5c67","Type":"ContainerStarted","Data":"f5d068162750213a772fd19bbd8dfa030abbd8a2ab41cf91bfb1758e389891ed"}
Sep 18 16:32:08.171549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.171527    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerID="e5b8adb1af139b7a1e0b0b52bc811bc0d8c343c01e0539d5f989c2725d53a981" exitCode=0
Sep 18 16:32:08.171643 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.171612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m" event={"ID":"fd319c98-354b-42c9-a105-bd3ec5a37a93","Type":"ContainerDied","Data":"e5b8adb1af139b7a1e0b0b52bc811bc0d8c343c01e0539d5f989c2725d53a981"}
Sep 18 16:32:08.171705 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.171648    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m" event={"ID":"fd319c98-354b-42c9-a105-bd3ec5a37a93","Type":"ContainerStarted","Data":"d7e9bf2717dca737e96c6c3ce531cb47cb4005565bf05345738c9e987337a423"}
Sep 18 16:32:08.173538 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.173517    2562 generic.go:356] "Generic (PLEG): container finished" podID="a4534922-1870-47d0-adae-0bffa332c0a2" containerID="8f647c02dd9fc0adb9811aebee13f5e147622809202d952eb0b0020a276560ec" exitCode=0
Sep 18 16:32:08.173627 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.173595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44" event={"ID":"a4534922-1870-47d0-adae-0bffa332c0a2","Type":"ContainerDied","Data":"8f647c02dd9fc0adb9811aebee13f5e147622809202d952eb0b0020a276560ec"}
Sep 18 16:32:08.173685 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:08.173635    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44" event={"ID":"a4534922-1870-47d0-adae-0bffa332c0a2","Type":"ContainerStarted","Data":"108a6aa96b53555729b3dbceb981796eec95917825c2d03cba7f3fc7e8adce83"}
Sep 18 16:32:10.187545 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.187502    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerID="60251f03930bae82be969f6d93a1d81890b7df66b974b7a321609d953133a5bc" exitCode=0
Sep 18 16:32:10.188010 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.187584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7" event={"ID":"bf71f09b-45b4-487d-8f35-5b7d884dc6ed","Type":"ContainerDied","Data":"60251f03930bae82be969f6d93a1d81890b7df66b974b7a321609d953133a5bc"}
Sep 18 16:32:10.189948 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.189919    2562 generic.go:356] "Generic (PLEG): container finished" podID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerID="5f403e76372729a4200da50e88edf881a580b475d09695533f06c96824c90c48" exitCode=0
Sep 18 16:32:10.190076 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.190018    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld" event={"ID":"89bffbfd-fd0d-4b40-bd83-376194bf5c67","Type":"ContainerDied","Data":"5f403e76372729a4200da50e88edf881a580b475d09695533f06c96824c90c48"}
Sep 18 16:32:10.192513 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.192488    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerID="9309f9d3444b8c38bf456daa9ae68e70ae1c0e8bb06779c90b6483f96c8d9bad" exitCode=0
Sep 18 16:32:10.192625 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.192585    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m" event={"ID":"fd319c98-354b-42c9-a105-bd3ec5a37a93","Type":"ContainerDied","Data":"9309f9d3444b8c38bf456daa9ae68e70ae1c0e8bb06779c90b6483f96c8d9bad"}
Sep 18 16:32:10.194731 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.194705    2562 generic.go:356] "Generic (PLEG): container finished" podID="a4534922-1870-47d0-adae-0bffa332c0a2" containerID="ebebc1292b870e9151e24cb3c147575ebcac03aa312a887489854593524741a8" exitCode=0
Sep 18 16:32:10.194811 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:10.194764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44" event={"ID":"a4534922-1870-47d0-adae-0bffa332c0a2","Type":"ContainerDied","Data":"ebebc1292b870e9151e24cb3c147575ebcac03aa312a887489854593524741a8"}
Sep 18 16:32:11.205660 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.205622    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerID="21d0f1c6fb96f6042e85348834b681e1498ec12645498526e1c429f71c38c70f" exitCode=0
Sep 18 16:32:11.206060 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.205704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7" event={"ID":"bf71f09b-45b4-487d-8f35-5b7d884dc6ed","Type":"ContainerDied","Data":"21d0f1c6fb96f6042e85348834b681e1498ec12645498526e1c429f71c38c70f"}
Sep 18 16:32:11.207992 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.207965    2562 generic.go:356] "Generic (PLEG): container finished" podID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerID="3967045d963494aafe9d923d1d97261a25e4e31c2d67b1fba7bf1b5327403d1f" exitCode=0
Sep 18 16:32:11.208139 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.208049    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld" event={"ID":"89bffbfd-fd0d-4b40-bd83-376194bf5c67","Type":"ContainerDied","Data":"3967045d963494aafe9d923d1d97261a25e4e31c2d67b1fba7bf1b5327403d1f"}
Sep 18 16:32:11.210309 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.210281    2562 generic.go:356] "Generic (PLEG): container finished" podID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerID="aa6593ef9eedef5f6c38dc061ffa6109e98e534222799a35f7526c57c72cc2a9" exitCode=0
Sep 18 16:32:11.210424 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.210366    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m" event={"ID":"fd319c98-354b-42c9-a105-bd3ec5a37a93","Type":"ContainerDied","Data":"aa6593ef9eedef5f6c38dc061ffa6109e98e534222799a35f7526c57c72cc2a9"}
Sep 18 16:32:11.212533 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.212508    2562 generic.go:356] "Generic (PLEG): container finished" podID="a4534922-1870-47d0-adae-0bffa332c0a2" containerID="d9cc7a0f87982685f1d9dd40961ccbee7364b7168178c2f8ea153178f22715fe" exitCode=0
Sep 18 16:32:11.212647 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:11.212582    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44" event={"ID":"a4534922-1870-47d0-adae-0bffa332c0a2","Type":"ContainerDied","Data":"d9cc7a0f87982685f1d9dd40961ccbee7364b7168178c2f8ea153178f22715fe"}
Sep 18 16:32:12.364222 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.364189    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:12.411204 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.411164    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:12.440589 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.440558    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:12.443574 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.443542    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:12.478042 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.477950    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54bxj\" (UniqueName: \"kubernetes.io/projected/a4534922-1870-47d0-adae-0bffa332c0a2-kube-api-access-54bxj\") pod \"a4534922-1870-47d0-adae-0bffa332c0a2\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") "
Sep 18 16:32:12.478042 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.478004    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-bundle\") pod \"a4534922-1870-47d0-adae-0bffa332c0a2\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") "
Sep 18 16:32:12.478289 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.478049    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-util\") pod \"a4534922-1870-47d0-adae-0bffa332c0a2\" (UID: \"a4534922-1870-47d0-adae-0bffa332c0a2\") "
Sep 18 16:32:12.478592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.478559    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-bundle" (OuterVolumeSpecName: "bundle") pod "a4534922-1870-47d0-adae-0bffa332c0a2" (UID: "a4534922-1870-47d0-adae-0bffa332c0a2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.480298 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.480266    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4534922-1870-47d0-adae-0bffa332c0a2-kube-api-access-54bxj" (OuterVolumeSpecName: "kube-api-access-54bxj") pod "a4534922-1870-47d0-adae-0bffa332c0a2" (UID: "a4534922-1870-47d0-adae-0bffa332c0a2"). InnerVolumeSpecName "kube-api-access-54bxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:32:12.487037 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.486995    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-util" (OuterVolumeSpecName: "util") pod "a4534922-1870-47d0-adae-0bffa332c0a2" (UID: "a4534922-1870-47d0-adae-0bffa332c0a2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.579520 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579476    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gsrzh\" (UniqueName: \"kubernetes.io/projected/89bffbfd-fd0d-4b40-bd83-376194bf5c67-kube-api-access-gsrzh\") pod \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") "
Sep 18 16:32:12.579520 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579518    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-bundle\") pod \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") "
Sep 18 16:32:12.579781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579569    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-util\") pod \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\" (UID: \"89bffbfd-fd0d-4b40-bd83-376194bf5c67\") "
Sep 18 16:32:12.579781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579591    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxkvc\" (UniqueName: \"kubernetes.io/projected/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-kube-api-access-vxkvc\") pod \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") "
Sep 18 16:32:12.579781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579613    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-bundle\") pod \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") "
Sep 18 16:32:12.579781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-bundle\") pod \"fd319c98-354b-42c9-a105-bd3ec5a37a93\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") "
Sep 18 16:32:12.579781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579656    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-util\") pod \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\" (UID: \"bf71f09b-45b4-487d-8f35-5b7d884dc6ed\") "
Sep 18 16:32:12.579781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579700    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbvc9\" (UniqueName: \"kubernetes.io/projected/fd319c98-354b-42c9-a105-bd3ec5a37a93-kube-api-access-sbvc9\") pod \"fd319c98-354b-42c9-a105-bd3ec5a37a93\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") "
Sep 18 16:32:12.579781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.579760    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-util\") pod \"fd319c98-354b-42c9-a105-bd3ec5a37a93\" (UID: \"fd319c98-354b-42c9-a105-bd3ec5a37a93\") "
Sep 18 16:32:12.580127 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.580011    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54bxj\" (UniqueName: \"kubernetes.io/projected/a4534922-1870-47d0-adae-0bffa332c0a2-kube-api-access-54bxj\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.580127 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.580040    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-bundle\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.580127 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.580073    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4534922-1870-47d0-adae-0bffa332c0a2-util\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.580652 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.580619    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-bundle" (OuterVolumeSpecName: "bundle") pod "bf71f09b-45b4-487d-8f35-5b7d884dc6ed" (UID: "bf71f09b-45b4-487d-8f35-5b7d884dc6ed"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.580845 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.580624    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-bundle" (OuterVolumeSpecName: "bundle") pod "89bffbfd-fd0d-4b40-bd83-376194bf5c67" (UID: "89bffbfd-fd0d-4b40-bd83-376194bf5c67"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.582254 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.582205    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-bundle" (OuterVolumeSpecName: "bundle") pod "fd319c98-354b-42c9-a105-bd3ec5a37a93" (UID: "fd319c98-354b-42c9-a105-bd3ec5a37a93"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.582254 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.582227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bffbfd-fd0d-4b40-bd83-376194bf5c67-kube-api-access-gsrzh" (OuterVolumeSpecName: "kube-api-access-gsrzh") pod "89bffbfd-fd0d-4b40-bd83-376194bf5c67" (UID: "89bffbfd-fd0d-4b40-bd83-376194bf5c67"). InnerVolumeSpecName "kube-api-access-gsrzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:32:12.582422 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.582387    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd319c98-354b-42c9-a105-bd3ec5a37a93-kube-api-access-sbvc9" (OuterVolumeSpecName: "kube-api-access-sbvc9") pod "fd319c98-354b-42c9-a105-bd3ec5a37a93" (UID: "fd319c98-354b-42c9-a105-bd3ec5a37a93"). InnerVolumeSpecName "kube-api-access-sbvc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:32:12.582591 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.582575    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-kube-api-access-vxkvc" (OuterVolumeSpecName: "kube-api-access-vxkvc") pod "bf71f09b-45b4-487d-8f35-5b7d884dc6ed" (UID: "bf71f09b-45b4-487d-8f35-5b7d884dc6ed"). InnerVolumeSpecName "kube-api-access-vxkvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:32:12.586580 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.586534    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-util" (OuterVolumeSpecName: "util") pod "fd319c98-354b-42c9-a105-bd3ec5a37a93" (UID: "fd319c98-354b-42c9-a105-bd3ec5a37a93"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.586697 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.586676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-util" (OuterVolumeSpecName: "util") pod "bf71f09b-45b4-487d-8f35-5b7d884dc6ed" (UID: "bf71f09b-45b4-487d-8f35-5b7d884dc6ed"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.586936 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.586914    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-util" (OuterVolumeSpecName: "util") pod "89bffbfd-fd0d-4b40-bd83-376194bf5c67" (UID: "89bffbfd-fd0d-4b40-bd83-376194bf5c67"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:32:12.680887 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680844    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-util\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.680887 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680877    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxkvc\" (UniqueName: \"kubernetes.io/projected/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-kube-api-access-vxkvc\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.680887 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680886    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-bundle\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.680887 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680894    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-bundle\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.681159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680903    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bf71f09b-45b4-487d-8f35-5b7d884dc6ed-util\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.681159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680912    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvc9\" (UniqueName: \"kubernetes.io/projected/fd319c98-354b-42c9-a105-bd3ec5a37a93-kube-api-access-sbvc9\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.681159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680920    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd319c98-354b-42c9-a105-bd3ec5a37a93-util\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.681159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680928    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsrzh\" (UniqueName: \"kubernetes.io/projected/89bffbfd-fd0d-4b40-bd83-376194bf5c67-kube-api-access-gsrzh\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:12.681159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:12.680937    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89bffbfd-fd0d-4b40-bd83-376194bf5c67-bundle\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:32:13.229746 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.229714    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7"
Sep 18 16:32:13.229938 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.229712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847amxjt7" event={"ID":"bf71f09b-45b4-487d-8f35-5b7d884dc6ed","Type":"ContainerDied","Data":"37312e532261fb05e63c00118e076d279d13bba2427fe522b2691491165759eb"}
Sep 18 16:32:13.229938 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.229885    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37312e532261fb05e63c00118e076d279d13bba2427fe522b2691491165759eb"
Sep 18 16:32:13.231758 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.231734    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld"
Sep 18 16:32:13.231882 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.231737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pklld" event={"ID":"89bffbfd-fd0d-4b40-bd83-376194bf5c67","Type":"ContainerDied","Data":"f5d068162750213a772fd19bbd8dfa030abbd8a2ab41cf91bfb1758e389891ed"}
Sep 18 16:32:13.231882 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.231841    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d068162750213a772fd19bbd8dfa030abbd8a2ab41cf91bfb1758e389891ed"
Sep 18 16:32:13.233725 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.233694    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m" event={"ID":"fd319c98-354b-42c9-a105-bd3ec5a37a93","Type":"ContainerDied","Data":"d7e9bf2717dca737e96c6c3ce531cb47cb4005565bf05345738c9e987337a423"}
Sep 18 16:32:13.233838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.233735    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7e9bf2717dca737e96c6c3ce531cb47cb4005565bf05345738c9e987337a423"
Sep 18 16:32:13.233838 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.233702    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c9wj5m"
Sep 18 16:32:13.235860 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.235842    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44"
Sep 18 16:32:13.235963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.235859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7j4q44" event={"ID":"a4534922-1870-47d0-adae-0bffa332c0a2","Type":"ContainerDied","Data":"108a6aa96b53555729b3dbceb981796eec95917825c2d03cba7f3fc7e8adce83"}
Sep 18 16:32:13.235963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:13.235903    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="108a6aa96b53555729b3dbceb981796eec95917825c2d03cba7f3fc7e8adce83"
Sep 18 16:32:20.651333 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651215    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"]
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651590    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651605    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651615    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4534922-1870-47d0-adae-0bffa332c0a2" containerName="extract"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651621    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4534922-1870-47d0-adae-0bffa332c0a2" containerName="extract"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651629    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651635    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651646    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4534922-1870-47d0-adae-0bffa332c0a2" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651651    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4534922-1870-47d0-adae-0bffa332c0a2" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651657    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651662    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651675    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651681    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651688    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651693    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerName="util"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651699    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4534922-1870-47d0-adae-0bffa332c0a2" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651704    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4534922-1870-47d0-adae-0bffa332c0a2" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651712    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651716    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerName="pull"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651724    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerName="extract"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651729    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerName="extract"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651736    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerName="extract"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651740    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerName="extract"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651748    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerName="extract"
Sep 18 16:32:20.651744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651754    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerName="extract"
Sep 18 16:32:20.652540 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651807    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd319c98-354b-42c9-a105-bd3ec5a37a93" containerName="extract"
Sep 18 16:32:20.652540 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651815    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf71f09b-45b4-487d-8f35-5b7d884dc6ed" containerName="extract"
Sep 18 16:32:20.652540 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651821    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="89bffbfd-fd0d-4b40-bd83-376194bf5c67" containerName="extract"
Sep 18 16:32:20.652540 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.651826    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4534922-1870-47d0-adae-0bffa332c0a2" containerName="extract"
Sep 18 16:32:20.656214 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.656189    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"
Sep 18 16:32:20.659173 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.659146    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 16:32:20.660087 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.660063    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-m2gdv\""
Sep 18 16:32:20.660155 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.660133    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 16:32:20.665897 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.665871    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"]
Sep 18 16:32:20.750651 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.750606    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jlxn\" (UniqueName: \"kubernetes.io/projected/72396fa1-0a6d-450f-ba7e-f44e6faf8f97-kube-api-access-6jlxn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-62kp7\" (UID: \"72396fa1-0a6d-450f-ba7e-f44e6faf8f97\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"
Sep 18 16:32:20.851085 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.851042    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jlxn\" (UniqueName: \"kubernetes.io/projected/72396fa1-0a6d-450f-ba7e-f44e6faf8f97-kube-api-access-6jlxn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-62kp7\" (UID: \"72396fa1-0a6d-450f-ba7e-f44e6faf8f97\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"
Sep 18 16:32:20.866827 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.866796    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jlxn\" (UniqueName: \"kubernetes.io/projected/72396fa1-0a6d-450f-ba7e-f44e6faf8f97-kube-api-access-6jlxn\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-62kp7\" (UID: \"72396fa1-0a6d-450f-ba7e-f44e6faf8f97\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"
Sep 18 16:32:20.967432 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:20.967339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"
Sep 18 16:32:21.259357 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:21.259269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"]
Sep 18 16:32:21.263905 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:32:21.263860    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72396fa1_0a6d_450f_ba7e_f44e6faf8f97.slice/crio-ee495e195afc7e56151452c35f7244a290cc6aa914d9c687b952f073019a0c81 WatchSource:0}: Error finding container ee495e195afc7e56151452c35f7244a290cc6aa914d9c687b952f073019a0c81: Status 404 returned error can't find the container with id ee495e195afc7e56151452c35f7244a290cc6aa914d9c687b952f073019a0c81
Sep 18 16:32:21.282154 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:21.282106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7" event={"ID":"72396fa1-0a6d-450f-ba7e-f44e6faf8f97","Type":"ContainerStarted","Data":"ee495e195afc7e56151452c35f7244a290cc6aa914d9c687b952f073019a0c81"}
Sep 18 16:32:23.298293 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:23.298249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7" event={"ID":"72396fa1-0a6d-450f-ba7e-f44e6faf8f97","Type":"ContainerStarted","Data":"6828f4255b51380d0ef9a2f68dd54cf560c28ff3a6f27ca7ec58ab2edfc384fb"}
Sep 18 16:32:23.299119 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:23.299090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"
Sep 18 16:32:27.772867 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.772808    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7" podStartSLOduration=6.192630978 podStartE2EDuration="7.772786633s" totalImagesPullingTime="1.580155655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:23.335019036 +0000 UTC m=+977.110451199" watchObservedRunningTime="2026-09-18 16:32:27.772786633 +0000 UTC m=+981.548218795"
Sep 18 16:32:27.773282 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.772948    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"]
Sep 18 16:32:27.776573 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.776540    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:27.779647 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.779619    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jskmc\""
Sep 18 16:32:27.798805 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.798756    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"]
Sep 18 16:32:27.816945 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.816886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpf65\" (UniqueName: \"kubernetes.io/projected/c1b524c7-f54c-467f-af5e-ce472c62c824-kube-api-access-lpf65\") pod \"kuadrant-operator-controller-manager-6d4cb79c-kpmzq\" (UID: \"c1b524c7-f54c-467f-af5e-ce472c62c824\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:27.817131 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.817029    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1b524c7-f54c-467f-af5e-ce472c62c824-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-kpmzq\" (UID: \"c1b524c7-f54c-467f-af5e-ce472c62c824\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:27.918176 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.918126    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpf65\" (UniqueName: \"kubernetes.io/projected/c1b524c7-f54c-467f-af5e-ce472c62c824-kube-api-access-lpf65\") pod \"kuadrant-operator-controller-manager-6d4cb79c-kpmzq\" (UID: \"c1b524c7-f54c-467f-af5e-ce472c62c824\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:27.918393 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.918228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1b524c7-f54c-467f-af5e-ce472c62c824-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-kpmzq\" (UID: \"c1b524c7-f54c-467f-af5e-ce472c62c824\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:27.918653 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.918630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1b524c7-f54c-467f-af5e-ce472c62c824-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-kpmzq\" (UID: \"c1b524c7-f54c-467f-af5e-ce472c62c824\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:27.939001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:27.938965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpf65\" (UniqueName: \"kubernetes.io/projected/c1b524c7-f54c-467f-af5e-ce472c62c824-kube-api-access-lpf65\") pod \"kuadrant-operator-controller-manager-6d4cb79c-kpmzq\" (UID: \"c1b524c7-f54c-467f-af5e-ce472c62c824\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:28.086407 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:28.086317    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:28.366837 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:28.366807    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"]
Sep 18 16:32:28.368887 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:32:28.368859    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b524c7_f54c_467f_af5e_ce472c62c824.slice/crio-e2bfaaf69eda82d3c00731e2c6a9cd89ebc8cb66065ed7e9636607836aa899c5 WatchSource:0}: Error finding container e2bfaaf69eda82d3c00731e2c6a9cd89ebc8cb66065ed7e9636607836aa899c5: Status 404 returned error can't find the container with id e2bfaaf69eda82d3c00731e2c6a9cd89ebc8cb66065ed7e9636607836aa899c5
Sep 18 16:32:29.337199 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:29.337165    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq" event={"ID":"c1b524c7-f54c-467f-af5e-ce472c62c824","Type":"ContainerStarted","Data":"e2bfaaf69eda82d3c00731e2c6a9cd89ebc8cb66065ed7e9636607836aa899c5"}
Sep 18 16:32:33.362911 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:33.362859    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq" event={"ID":"c1b524c7-f54c-467f-af5e-ce472c62c824","Type":"ContainerStarted","Data":"383ad5871bde2b7363a30172b9a7c857de2eccb9f2149957d985220de3d32f6e"}
Sep 18 16:32:33.363317 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:33.362993    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:32:33.386655 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:33.386597    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq" podStartSLOduration=2.4275939060000002 podStartE2EDuration="6.38658047s" totalImagesPullingTime="3.958986564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:32:33.384273629 +0000 UTC m=+987.159705815" watchObservedRunningTime="2026-09-18 16:32:33.38658047 +0000 UTC m=+987.162012628"
Sep 18 16:32:35.312114 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:35.312083    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-62kp7"
Sep 18 16:32:44.370569 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:32:44.370527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-kpmzq"
Sep 18 16:33:19.996549 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:19.996510    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fbm42"]
Sep 18 16:33:19.999779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:19.999750    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.002724 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.002692    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 16:33:20.002881 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.002698    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-24526\""
Sep 18 16:33:20.010580 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.010553    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fbm42"]
Sep 18 16:33:20.040535 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.040496    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fbm42"]
Sep 18 16:33:20.064528 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.064486    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ff20ff04-e532-4e19-a755-62ed27751afa-config-file\") pod \"limitador-limitador-84994f7ffc-fbm42\" (UID: \"ff20ff04-e532-4e19-a755-62ed27751afa\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.064714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.064565    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrn9\" (UniqueName: \"kubernetes.io/projected/ff20ff04-e532-4e19-a755-62ed27751afa-kube-api-access-zvrn9\") pod \"limitador-limitador-84994f7ffc-fbm42\" (UID: \"ff20ff04-e532-4e19-a755-62ed27751afa\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.165367 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.165320    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrn9\" (UniqueName: \"kubernetes.io/projected/ff20ff04-e532-4e19-a755-62ed27751afa-kube-api-access-zvrn9\") pod \"limitador-limitador-84994f7ffc-fbm42\" (UID: \"ff20ff04-e532-4e19-a755-62ed27751afa\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.165550 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.165388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ff20ff04-e532-4e19-a755-62ed27751afa-config-file\") pod \"limitador-limitador-84994f7ffc-fbm42\" (UID: \"ff20ff04-e532-4e19-a755-62ed27751afa\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.166022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.166001    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ff20ff04-e532-4e19-a755-62ed27751afa-config-file\") pod \"limitador-limitador-84994f7ffc-fbm42\" (UID: \"ff20ff04-e532-4e19-a755-62ed27751afa\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.176661 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.176622    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrn9\" (UniqueName: \"kubernetes.io/projected/ff20ff04-e532-4e19-a755-62ed27751afa-kube-api-access-zvrn9\") pod \"limitador-limitador-84994f7ffc-fbm42\" (UID: \"ff20ff04-e532-4e19-a755-62ed27751afa\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.311952 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.311817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:20.591860 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.591829    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-fbm42"]
Sep 18 16:33:20.593786 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:33:20.593753    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff20ff04_e532_4e19_a755_62ed27751afa.slice/crio-73bb200d57d75315ad82fbe2dfbfe070194098ecf35fe6a6ee320e0e5e7dd831 WatchSource:0}: Error finding container 73bb200d57d75315ad82fbe2dfbfe070194098ecf35fe6a6ee320e0e5e7dd831: Status 404 returned error can't find the container with id 73bb200d57d75315ad82fbe2dfbfe070194098ecf35fe6a6ee320e0e5e7dd831
Sep 18 16:33:20.612960 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:20.612918    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42" event={"ID":"ff20ff04-e532-4e19-a755-62ed27751afa","Type":"ContainerStarted","Data":"73bb200d57d75315ad82fbe2dfbfe070194098ecf35fe6a6ee320e0e5e7dd831"}
Sep 18 16:33:22.626688 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:22.626645    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42" event={"ID":"ff20ff04-e532-4e19-a755-62ed27751afa","Type":"ContainerStarted","Data":"8120756bbcbed7c9d7ccc922cfb7494d953c927cf6dffbe263ec69189ee89142"}
Sep 18 16:33:22.627084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:22.626747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:33:22.643126 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:22.643062    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42" podStartSLOduration=1.876164868 podStartE2EDuration="3.643041943s" totalImagesPullingTime="1.766877075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:33:22.6420631 +0000 UTC m=+1036.417495276" watchObservedRunningTime="2026-09-18 16:33:22.643041943 +0000 UTC m=+1036.418474104"
Sep 18 16:33:33.633539 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:33:33.633507    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-fbm42"
Sep 18 16:34:02.148037 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.147997    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"]
Sep 18 16:34:02.151196 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.151168    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.169674 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.169637    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"]
Sep 18 16:34:02.232159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.231669    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6pb\" (UniqueName: \"kubernetes.io/projected/0c9e11c8-f06a-4418-845e-7261062359ec-kube-api-access-vp6pb\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.232159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.231714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9e11c8-f06a-4418-845e-7261062359ec-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.232159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.231733    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9e11c8-f06a-4418-845e-7261062359ec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.232159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.231810    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.232159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.231844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9e11c8-f06a-4418-845e-7261062359ec-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.232159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.231866    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.232159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.231882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.332486 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.332445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6pb\" (UniqueName: \"kubernetes.io/projected/0c9e11c8-f06a-4418-845e-7261062359ec-kube-api-access-vp6pb\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.332729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.332512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9e11c8-f06a-4418-845e-7261062359ec-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.332729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.332596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9e11c8-f06a-4418-845e-7261062359ec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.332729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.332641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.332729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.332675    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9e11c8-f06a-4418-845e-7261062359ec-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.332729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.332705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.333073 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.332750    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.333714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.333686    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0c9e11c8-f06a-4418-845e-7261062359ec-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.335989 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.335900    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9e11c8-f06a-4418-845e-7261062359ec-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.335989 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.335909    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.335989 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.335953    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.336226 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.336002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0c9e11c8-f06a-4418-845e-7261062359ec-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.347907 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.347866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6pb\" (UniqueName: \"kubernetes.io/projected/0c9e11c8-f06a-4418-845e-7261062359ec-kube-api-access-vp6pb\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.355620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.355582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9e11c8-f06a-4418-845e-7261062359ec-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-r2wkt\" (UID: \"0c9e11c8-f06a-4418-845e-7261062359ec\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.460707 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.460618    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:02.756292 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.756269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"]
Sep 18 16:34:02.758417 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:34:02.758391    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9e11c8_f06a_4418_845e_7261062359ec.slice/crio-e0dadb4ed9f255fa637ea54e3a6be30d12bcc64cd1c5ae9e42ecf0359ae81d9c WatchSource:0}: Error finding container e0dadb4ed9f255fa637ea54e3a6be30d12bcc64cd1c5ae9e42ecf0359ae81d9c: Status 404 returned error can't find the container with id e0dadb4ed9f255fa637ea54e3a6be30d12bcc64cd1c5ae9e42ecf0359ae81d9c
Sep 18 16:34:02.760649 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.760618    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 16:34:02.760726 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.760677    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 16:34:02.841829 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:02.841795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt" event={"ID":"0c9e11c8-f06a-4418-845e-7261062359ec","Type":"ContainerStarted","Data":"e0dadb4ed9f255fa637ea54e3a6be30d12bcc64cd1c5ae9e42ecf0359ae81d9c"}
Sep 18 16:34:03.854902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:03.854863    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt" event={"ID":"0c9e11c8-f06a-4418-845e-7261062359ec","Type":"ContainerStarted","Data":"d8c0015b7759dceddb2374c5345183d9d19f1dd106666867cd45285b5bff0ca6"}
Sep 18 16:34:03.855417 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:03.855031    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:03.856995 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:03.856967    2562 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-r2wkt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 16:34:03.857109 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:03.857017    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt" podUID="0c9e11c8-f06a-4418-845e-7261062359ec" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 16:34:03.900299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:03.900221    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt" podStartSLOduration=1.900203065 podStartE2EDuration="1.900203065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:34:03.89791137 +0000 UTC m=+1077.673343546" watchObservedRunningTime="2026-09-18 16:34:03.900203065 +0000 UTC m=+1077.675635226"
Sep 18 16:34:04.861720 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:04.861690    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-r2wkt"
Sep 18 16:34:04.914948 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:04.914906    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"]
Sep 18 16:34:04.915132 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:04.915110    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr" podUID="9eb48228-6af3-420e-b596-620dab4086c8" containerName="discovery" containerID="cri-o://81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc" gracePeriod=30
Sep 18 16:34:05.181125 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.181098    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:34:05.362466 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.362433    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-cacerts\") pod \"9eb48228-6af3-420e-b596-620dab4086c8\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") "
Sep 18 16:34:05.362654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.362479    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb48228-6af3-420e-b596-620dab4086c8-local-certs\") pod \"9eb48228-6af3-420e-b596-620dab4086c8\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") "
Sep 18 16:34:05.362654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.362499    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-dns-cert\") pod \"9eb48228-6af3-420e-b596-620dab4086c8\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") "
Sep 18 16:34:05.362654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.362531    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ggsv\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-kube-api-access-7ggsv\") pod \"9eb48228-6af3-420e-b596-620dab4086c8\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") "
Sep 18 16:34:05.362654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.362553    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-istio-token\") pod \"9eb48228-6af3-420e-b596-620dab4086c8\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") "
Sep 18 16:34:05.362654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.362599    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-kubeconfig\") pod \"9eb48228-6af3-420e-b596-620dab4086c8\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") "
Sep 18 16:34:05.362654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.362625    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-ca-configmap\") pod \"9eb48228-6af3-420e-b596-620dab4086c8\" (UID: \"9eb48228-6af3-420e-b596-620dab4086c8\") "
Sep 18 16:34:05.363209 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.363154    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9eb48228-6af3-420e-b596-620dab4086c8" (UID: "9eb48228-6af3-420e-b596-620dab4086c8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:34:05.365334 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.365289    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9eb48228-6af3-420e-b596-620dab4086c8" (UID: "9eb48228-6af3-420e-b596-620dab4086c8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:05.365466 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.365384    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9eb48228-6af3-420e-b596-620dab4086c8" (UID: "9eb48228-6af3-420e-b596-620dab4086c8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:05.365466 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.365416    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-cacerts" (OuterVolumeSpecName: "cacerts") pod "9eb48228-6af3-420e-b596-620dab4086c8" (UID: "9eb48228-6af3-420e-b596-620dab4086c8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:34:05.365576 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.365532    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "9eb48228-6af3-420e-b596-620dab4086c8" (UID: "9eb48228-6af3-420e-b596-620dab4086c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:05.365669 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.365646    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb48228-6af3-420e-b596-620dab4086c8-local-certs" (OuterVolumeSpecName: "local-certs") pod "9eb48228-6af3-420e-b596-620dab4086c8" (UID: "9eb48228-6af3-420e-b596-620dab4086c8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:34:05.365729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.365688    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-kube-api-access-7ggsv" (OuterVolumeSpecName: "kube-api-access-7ggsv") pod "9eb48228-6af3-420e-b596-620dab4086c8" (UID: "9eb48228-6af3-420e-b596-620dab4086c8"). InnerVolumeSpecName "kube-api-access-7ggsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:34:05.464326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.464187    2562 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-cacerts\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:34:05.464326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.464228    2562 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb48228-6af3-420e-b596-620dab4086c8-local-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:34:05.464326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.464276    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-dns-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:34:05.464326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.464289    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ggsv\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-kube-api-access-7ggsv\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:34:05.464326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.464304    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb48228-6af3-420e-b596-620dab4086c8-istio-token\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:34:05.464326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.464318    2562 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9eb48228-6af3-420e-b596-620dab4086c8-istio-kubeconfig\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:34:05.464326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.464331    2562 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9eb48228-6af3-420e-b596-620dab4086c8-istio-csr-ca-configmap\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:34:05.868166 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.868130    2562 generic.go:356] "Generic (PLEG): container finished" podID="9eb48228-6af3-420e-b596-620dab4086c8" containerID="81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc" exitCode=0
Sep 18 16:34:05.868624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.868179    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"
Sep 18 16:34:05.868624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.868207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr" event={"ID":"9eb48228-6af3-420e-b596-620dab4086c8","Type":"ContainerDied","Data":"81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc"}
Sep 18 16:34:05.868624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.868264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr" event={"ID":"9eb48228-6af3-420e-b596-620dab4086c8","Type":"ContainerDied","Data":"93f5a594ee5277babf08d3d7957a5e97f5da49e6cdbc660ee7983d4d48160ef0"}
Sep 18 16:34:05.868624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.868297    2562 scope.go:117] "RemoveContainer" containerID="81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc"
Sep 18 16:34:05.878468 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.878441    2562 scope.go:117] "RemoveContainer" containerID="81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc"
Sep 18 16:34:05.878824 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:34:05.878802    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc\": container with ID starting with 81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc not found: ID does not exist" containerID="81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc"
Sep 18 16:34:05.878904 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.878834    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc"} err="failed to get container status \"81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc\": rpc error: code = NotFound desc = could not find container \"81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc\": container with ID starting with 81c344a1c3898ac59e3835fc336af35ad74edd055831e80c1b45f186729657bc not found: ID does not exist"
Sep 18 16:34:05.891107 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.891070    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"]
Sep 18 16:34:05.894846 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:05.894812    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ggmfr"]
Sep 18 16:34:06.335037 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:34:06.334951    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb48228-6af3-420e-b596-620dab4086c8" path="/var/lib/kubelet/pods/9eb48228-6af3-420e-b596-620dab4086c8/volumes"
Sep 18 16:43:23.190487 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.190444    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"]
Sep 18 16:43:23.191978 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.190851    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb48228-6af3-420e-b596-620dab4086c8" containerName="discovery"
Sep 18 16:43:23.191978 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.190864    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb48228-6af3-420e-b596-620dab4086c8" containerName="discovery"
Sep 18 16:43:23.191978 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.190925    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb48228-6af3-420e-b596-620dab4086c8" containerName="discovery"
Sep 18 16:43:23.192906 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.192890    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.196866 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.196820    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 18 16:43:23.196866 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.196821    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-xp2ct\""
Sep 18 16:43:23.197107 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.196821    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 18 16:43:23.197107 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.196821    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 18 16:43:23.203098 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.203069    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"]
Sep 18 16:43:23.277782 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.277749    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-dshm\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.277976 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.277804    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-home\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.277976 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.277827    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.277976 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.277882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl55m\" (UniqueName: \"kubernetes.io/projected/959d28b2-2d91-4730-bc78-b84cf87260af-kube-api-access-sl55m\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.277976 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.277907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.277976 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.277967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d28b2-2d91-4730-bc78-b84cf87260af-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.378764 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.378723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-dshm\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.378950 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.378799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-home\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.378950 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.378827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.378950 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.378887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl55m\" (UniqueName: \"kubernetes.io/projected/959d28b2-2d91-4730-bc78-b84cf87260af-kube-api-access-sl55m\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.378950 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.378922    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.379188 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.378956    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d28b2-2d91-4730-bc78-b84cf87260af-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.379369 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.379338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.379603 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.379562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.379679 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.379623    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-home\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.381227 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.381204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-dshm\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.381627 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.381606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d28b2-2d91-4730-bc78-b84cf87260af-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.390117 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.390084    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl55m\" (UniqueName: \"kubernetes.io/projected/959d28b2-2d91-4730-bc78-b84cf87260af-kube-api-access-sl55m\") pod \"fc-smoke-test-kserve-7f4687897-n4rnq\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.505145 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.505036    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:23.782958 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.782926    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"]
Sep 18 16:43:23.784895 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:43:23.784865    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959d28b2_2d91_4730_bc78_b84cf87260af.slice/crio-6e2a49d2e40c587a5f4cf18c02f5cc7689e0b9d03d3e25997129d9ede1f0e664 WatchSource:0}: Error finding container 6e2a49d2e40c587a5f4cf18c02f5cc7689e0b9d03d3e25997129d9ede1f0e664: Status 404 returned error can't find the container with id 6e2a49d2e40c587a5f4cf18c02f5cc7689e0b9d03d3e25997129d9ede1f0e664
Sep 18 16:43:23.786932 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.786912    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 16:43:23.894987 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:23.894947    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" event={"ID":"959d28b2-2d91-4730-bc78-b84cf87260af","Type":"ContainerStarted","Data":"6e2a49d2e40c587a5f4cf18c02f5cc7689e0b9d03d3e25997129d9ede1f0e664"}
Sep 18 16:43:25.908839 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:25.908804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" event={"ID":"959d28b2-2d91-4730-bc78-b84cf87260af","Type":"ContainerStarted","Data":"833e81d16d7a34c007f8411f568ab099e29dd9e65283c78c9cbf4e220b8e23e6"}
Sep 18 16:43:25.927299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:25.927220    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" podStartSLOduration=1.193433107 podStartE2EDuration="2.927200652s" totalImagesPullingTime="1.733767545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:43:25.925120664 +0000 UTC m=+1639.700552841" watchObservedRunningTime="2026-09-18 16:43:25.927200652 +0000 UTC m=+1639.702632861"
Sep 18 16:43:26.506061 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:26.506021    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:33.505904 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:33.505857    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:33.519122 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:33.519092    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:43:33.963364 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:43:33.963332    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:44:02.773903 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:02.773866    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"]
Sep 18 16:44:02.783557 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:44:02.783502    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:44:02.783557 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:44:02.783557 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:44:02.783557 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="833e81d16d7a34c007f8411f568ab099e29dd9e65283c78c9cbf4e220b8e23e6" cmd=["/bin/sleep","15"]
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:02.783584    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" message="" err=<
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:44:02.783597    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" podUID="959d28b2-2d91-4730-bc78-b84cf87260af" containerName="main" containerID="cri-o://833e81d16d7a34c007f8411f568ab099e29dd9e65283c78c9cbf4e220b8e23e6"
Sep 18 16:44:02.783967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:02.783649    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" podUID="959d28b2-2d91-4730-bc78-b84cf87260af" containerName="main" containerID="cri-o://833e81d16d7a34c007f8411f568ab099e29dd9e65283c78c9cbf4e220b8e23e6" gracePeriod=60
Sep 18 16:44:02.953173 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:02.953131    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" podUID="959d28b2-2d91-4730-bc78-b84cf87260af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Sep 18 16:44:03.119576 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.119541    2562 generic.go:356] "Generic (PLEG): container finished" podID="959d28b2-2d91-4730-bc78-b84cf87260af" containerID="833e81d16d7a34c007f8411f568ab099e29dd9e65283c78c9cbf4e220b8e23e6" exitCode=0
Sep 18 16:44:03.119782 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.119620    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" event={"ID":"959d28b2-2d91-4730-bc78-b84cf87260af","Type":"ContainerDied","Data":"833e81d16d7a34c007f8411f568ab099e29dd9e65283c78c9cbf4e220b8e23e6"}
Sep 18 16:44:03.146787 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.146763    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:44:03.204580 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204550    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-model-cache\") pod \"959d28b2-2d91-4730-bc78-b84cf87260af\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") "
Sep 18 16:44:03.204779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-tmp-dir\") pod \"959d28b2-2d91-4730-bc78-b84cf87260af\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") "
Sep 18 16:44:03.204779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-home\") pod \"959d28b2-2d91-4730-bc78-b84cf87260af\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") "
Sep 18 16:44:03.204779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204642    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl55m\" (UniqueName: \"kubernetes.io/projected/959d28b2-2d91-4730-bc78-b84cf87260af-kube-api-access-sl55m\") pod \"959d28b2-2d91-4730-bc78-b84cf87260af\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") "
Sep 18 16:44:03.204779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204664    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d28b2-2d91-4730-bc78-b84cf87260af-tls-certs\") pod \"959d28b2-2d91-4730-bc78-b84cf87260af\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") "
Sep 18 16:44:03.204779 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204708    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-dshm\") pod \"959d28b2-2d91-4730-bc78-b84cf87260af\" (UID: \"959d28b2-2d91-4730-bc78-b84cf87260af\") "
Sep 18 16:44:03.205046 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204903    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-home" (OuterVolumeSpecName: "home") pod "959d28b2-2d91-4730-bc78-b84cf87260af" (UID: "959d28b2-2d91-4730-bc78-b84cf87260af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:03.205046 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204915    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "959d28b2-2d91-4730-bc78-b84cf87260af" (UID: "959d28b2-2d91-4730-bc78-b84cf87260af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:03.205046 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.204927    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-model-cache" (OuterVolumeSpecName: "model-cache") pod "959d28b2-2d91-4730-bc78-b84cf87260af" (UID: "959d28b2-2d91-4730-bc78-b84cf87260af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:03.206967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.206932    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/959d28b2-2d91-4730-bc78-b84cf87260af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "959d28b2-2d91-4730-bc78-b84cf87260af" (UID: "959d28b2-2d91-4730-bc78-b84cf87260af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:44:03.206967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.206948    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-dshm" (OuterVolumeSpecName: "dshm") pod "959d28b2-2d91-4730-bc78-b84cf87260af" (UID: "959d28b2-2d91-4730-bc78-b84cf87260af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:44:03.207157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.207077    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/959d28b2-2d91-4730-bc78-b84cf87260af-kube-api-access-sl55m" (OuterVolumeSpecName: "kube-api-access-sl55m") pod "959d28b2-2d91-4730-bc78-b84cf87260af" (UID: "959d28b2-2d91-4730-bc78-b84cf87260af"). InnerVolumeSpecName "kube-api-access-sl55m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:44:03.305592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.305491    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959d28b2-2d91-4730-bc78-b84cf87260af-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:44:03.305592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.305535    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:44:03.305592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.305544    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:44:03.305592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.305553    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:44:03.305592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.305561    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959d28b2-2d91-4730-bc78-b84cf87260af-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:44:03.305592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:03.305569    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl55m\" (UniqueName: \"kubernetes.io/projected/959d28b2-2d91-4730-bc78-b84cf87260af-kube-api-access-sl55m\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:44:04.127400 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:04.127367    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"
Sep 18 16:44:04.127400 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:04.127386    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq" event={"ID":"959d28b2-2d91-4730-bc78-b84cf87260af","Type":"ContainerDied","Data":"6e2a49d2e40c587a5f4cf18c02f5cc7689e0b9d03d3e25997129d9ede1f0e664"}
Sep 18 16:44:04.127942 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:04.127464    2562 scope.go:117] "RemoveContainer" containerID="833e81d16d7a34c007f8411f568ab099e29dd9e65283c78c9cbf4e220b8e23e6"
Sep 18 16:44:04.151493 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:04.151458    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"]
Sep 18 16:44:04.155830 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:04.155804    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-n4rnq"]
Sep 18 16:44:04.335556 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:04.335512    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="959d28b2-2d91-4730-bc78-b84cf87260af" path="/var/lib/kubelet/pods/959d28b2-2d91-4730-bc78-b84cf87260af/volumes"
Sep 18 16:44:30.033034 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.032996    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"]
Sep 18 16:44:30.033458 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.033443    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="959d28b2-2d91-4730-bc78-b84cf87260af" containerName="main"
Sep 18 16:44:30.033500 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.033460    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="959d28b2-2d91-4730-bc78-b84cf87260af" containerName="main"
Sep 18 16:44:30.033536 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.033527    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="959d28b2-2d91-4730-bc78-b84cf87260af" containerName="main"
Sep 18 16:44:30.036577 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.036555    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.039251 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.039216    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-t4nsz\""
Sep 18 16:44:30.040149 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.040127    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 18 16:44:30.040298 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.040203    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 18 16:44:30.040298 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.040204    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 18 16:44:30.040419 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.040325    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-gnnmb\""
Sep 18 16:44:30.049904 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.049878    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"]
Sep 18 16:44:30.139769 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.139718    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.139963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.139790    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.139963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.139818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.139963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.139840    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.139963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.139919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpr4g\" (UniqueName: \"kubernetes.io/projected/06028d51-c9d4-48fb-9846-5c024cc9480f-kube-api-access-jpr4g\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.139963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.139963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06028d51-c9d4-48fb-9846-5c024cc9480f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241232 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241431 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpr4g\" (UniqueName: \"kubernetes.io/projected/06028d51-c9d4-48fb-9846-5c024cc9480f-kube-api-access-jpr4g\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241431 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241332    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06028d51-c9d4-48fb-9846-5c024cc9480f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241431 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241376    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241431 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241625 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241685 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241651    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241785 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241757    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.241843 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.241811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.243770 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.243749    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.244067 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.244050    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06028d51-c9d4-48fb-9846-5c024cc9480f-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.249874 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.249844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpr4g\" (UniqueName: \"kubernetes.io/projected/06028d51-c9d4-48fb-9846-5c024cc9480f-kube-api-access-jpr4g\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.347088 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.346469    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:30.623182 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:30.623153    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"]
Sep 18 16:44:30.624832 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:44:30.624805    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06028d51_c9d4_48fb_9846_5c024cc9480f.slice/crio-2170ab7def548f6050180d7f3c2996b8f894eabab2a87c35bd4ca0651831277e WatchSource:0}: Error finding container 2170ab7def548f6050180d7f3c2996b8f894eabab2a87c35bd4ca0651831277e: Status 404 returned error can't find the container with id 2170ab7def548f6050180d7f3c2996b8f894eabab2a87c35bd4ca0651831277e
Sep 18 16:44:31.270733 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:31.270682    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" event={"ID":"06028d51-c9d4-48fb-9846-5c024cc9480f","Type":"ContainerStarted","Data":"2170ab7def548f6050180d7f3c2996b8f894eabab2a87c35bd4ca0651831277e"}
Sep 18 16:44:33.283861 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:33.283753    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" event={"ID":"06028d51-c9d4-48fb-9846-5c024cc9480f","Type":"ContainerStarted","Data":"9943c8367379221f96544d7f5e53ba1d8413df5a8185bac3cc48bc907cf68e5b"}
Sep 18 16:44:33.284223 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:33.283892    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:34.291536 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:34.291494    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" event={"ID":"06028d51-c9d4-48fb-9846-5c024cc9480f","Type":"ContainerStarted","Data":"9b01d8a81e51bf2ac75dc78d988cc2ce322e20dc393d648f0b0108a727233ad3"}
Sep 18 16:44:34.313256 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:34.313195    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" podStartSLOduration=2.070800173 podStartE2EDuration="4.313176516s" totalImagesPullingTime="2.242376343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:44:34.3109253 +0000 UTC m=+1708.086357494" watchObservedRunningTime="2026-09-18 16:44:34.313176516 +0000 UTC m=+1708.088608677"
Sep 18 16:44:34.347686 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:34.347648    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:40.348024 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:40.347985    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:40.360486 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:40.360459    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:41.345013 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:41.344980    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:44:42.349790 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:44:42.349760    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:46:19.526646 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.526609    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"]
Sep 18 16:46:19.537539 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:46:19.537490    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:46:19.537539 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:46:19.537539 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:46:19.537539 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="9b01d8a81e51bf2ac75dc78d988cc2ce322e20dc393d648f0b0108a727233ad3" cmd=["/bin/sleep","15"]
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.537551    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" message="" err=<
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:46:19.537568    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="main" containerID="cri-o://9b01d8a81e51bf2ac75dc78d988cc2ce322e20dc393d648f0b0108a727233ad3"
Sep 18 16:46:19.537814 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.537613    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="main" containerID="cri-o://9b01d8a81e51bf2ac75dc78d988cc2ce322e20dc393d648f0b0108a727233ad3" gracePeriod=60
Sep 18 16:46:19.754256 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.754197    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="llm-d-routing-sidecar" containerID="cri-o://9943c8367379221f96544d7f5e53ba1d8413df5a8185bac3cc48bc907cf68e5b" gracePeriod=60
Sep 18 16:46:19.886509 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.886475    2562 generic.go:356] "Generic (PLEG): container finished" podID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerID="9b01d8a81e51bf2ac75dc78d988cc2ce322e20dc393d648f0b0108a727233ad3" exitCode=0
Sep 18 16:46:19.886509 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.886503    2562 generic.go:356] "Generic (PLEG): container finished" podID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerID="9943c8367379221f96544d7f5e53ba1d8413df5a8185bac3cc48bc907cf68e5b" exitCode=0
Sep 18 16:46:19.886716 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.886554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" event={"ID":"06028d51-c9d4-48fb-9846-5c024cc9480f","Type":"ContainerDied","Data":"9b01d8a81e51bf2ac75dc78d988cc2ce322e20dc393d648f0b0108a727233ad3"}
Sep 18 16:46:19.886716 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:19.886634    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" event={"ID":"06028d51-c9d4-48fb-9846-5c024cc9480f","Type":"ContainerDied","Data":"9943c8367379221f96544d7f5e53ba1d8413df5a8185bac3cc48bc907cf68e5b"}
Sep 18 16:46:20.007373 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.007343    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:46:20.080899 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.080864    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06028d51-c9d4-48fb-9846-5c024cc9480f-tls-certs\") pod \"06028d51-c9d4-48fb-9846-5c024cc9480f\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") "
Sep 18 16:46:20.081097 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.080926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-dshm\") pod \"06028d51-c9d4-48fb-9846-5c024cc9480f\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") "
Sep 18 16:46:20.081097 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.080949    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-model-cache\") pod \"06028d51-c9d4-48fb-9846-5c024cc9480f\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") "
Sep 18 16:46:20.081097 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.080968    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpr4g\" (UniqueName: \"kubernetes.io/projected/06028d51-c9d4-48fb-9846-5c024cc9480f-kube-api-access-jpr4g\") pod \"06028d51-c9d4-48fb-9846-5c024cc9480f\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") "
Sep 18 16:46:20.081097 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.081019    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-home\") pod \"06028d51-c9d4-48fb-9846-5c024cc9480f\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") "
Sep 18 16:46:20.081097 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.081041    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-tmp-dir\") pod \"06028d51-c9d4-48fb-9846-5c024cc9480f\" (UID: \"06028d51-c9d4-48fb-9846-5c024cc9480f\") "
Sep 18 16:46:20.081392 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.081340    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-home" (OuterVolumeSpecName: "home") pod "06028d51-c9d4-48fb-9846-5c024cc9480f" (UID: "06028d51-c9d4-48fb-9846-5c024cc9480f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.081392 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.081352    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-model-cache" (OuterVolumeSpecName: "model-cache") pod "06028d51-c9d4-48fb-9846-5c024cc9480f" (UID: "06028d51-c9d4-48fb-9846-5c024cc9480f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.081471 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.081389    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06028d51-c9d4-48fb-9846-5c024cc9480f" (UID: "06028d51-c9d4-48fb-9846-5c024cc9480f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.083372 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.083343    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06028d51-c9d4-48fb-9846-5c024cc9480f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06028d51-c9d4-48fb-9846-5c024cc9480f" (UID: "06028d51-c9d4-48fb-9846-5c024cc9480f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:46:20.083614 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.083592    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06028d51-c9d4-48fb-9846-5c024cc9480f-kube-api-access-jpr4g" (OuterVolumeSpecName: "kube-api-access-jpr4g") pod "06028d51-c9d4-48fb-9846-5c024cc9480f" (UID: "06028d51-c9d4-48fb-9846-5c024cc9480f"). InnerVolumeSpecName "kube-api-access-jpr4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:46:20.083719 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.083652    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-dshm" (OuterVolumeSpecName: "dshm") pod "06028d51-c9d4-48fb-9846-5c024cc9480f" (UID: "06028d51-c9d4-48fb-9846-5c024cc9480f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:46:20.181912 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.181819    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.181912 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.181850    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.181912 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.181861    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06028d51-c9d4-48fb-9846-5c024cc9480f-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.181912 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.181869    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.181912 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.181877    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06028d51-c9d4-48fb-9846-5c024cc9480f-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.181912 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.181886    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpr4g\" (UniqueName: \"kubernetes.io/projected/06028d51-c9d4-48fb-9846-5c024cc9480f-kube-api-access-jpr4g\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:46:20.894230 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.894198    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"
Sep 18 16:46:20.894230 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.894206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4" event={"ID":"06028d51-c9d4-48fb-9846-5c024cc9480f","Type":"ContainerDied","Data":"2170ab7def548f6050180d7f3c2996b8f894eabab2a87c35bd4ca0651831277e"}
Sep 18 16:46:20.894740 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.894295    2562 scope.go:117] "RemoveContainer" containerID="9b01d8a81e51bf2ac75dc78d988cc2ce322e20dc393d648f0b0108a727233ad3"
Sep 18 16:46:20.903898 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.903876    2562 scope.go:117] "RemoveContainer" containerID="9943c8367379221f96544d7f5e53ba1d8413df5a8185bac3cc48bc907cf68e5b"
Sep 18 16:46:20.914795 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.914763    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"]
Sep 18 16:46:20.918880 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:20.918850    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-mqqd4"]
Sep 18 16:46:22.334554 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:22.334516    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" path="/var/lib/kubelet/pods/06028d51-c9d4-48fb-9846-5c024cc9480f/volumes"
Sep 18 16:46:58.503857 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.503816    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"]
Sep 18 16:46:58.504411 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.504390    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="llm-d-routing-sidecar"
Sep 18 16:46:58.504491 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.504413    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="llm-d-routing-sidecar"
Sep 18 16:46:58.504491 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.504449    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="main"
Sep 18 16:46:58.504491 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.504458    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="main"
Sep 18 16:46:58.504637 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.504549    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="llm-d-routing-sidecar"
Sep 18 16:46:58.504637 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.504565    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="06028d51-c9d4-48fb-9846-5c024cc9480f" containerName="main"
Sep 18 16:46:58.507026 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.507000    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.510094 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.510067    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Sep 18 16:46:58.510857 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.510839    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Sep 18 16:46:58.510963 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.510856    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Sep 18 16:46:58.513792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.513765    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-w9swt\""
Sep 18 16:46:58.532842 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.532803    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"]
Sep 18 16:46:58.602299 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602254    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-credential-socket\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602501 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-podinfo\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602501 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-token\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602501 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-socket\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602501 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602405    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-data\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602501 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602488    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5q9\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-kube-api-access-2q5q9\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602748 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602540    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-certs\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602748 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602582    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.602748 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.602616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-envoy\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.703542 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.703542 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703548    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-envoy\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.703783 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-credential-socket\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.703783 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-podinfo\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.703783 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703772    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-token\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703801    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-socket\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703849    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-data\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703892    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5q9\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-kube-api-access-2q5q9\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-certs\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.703978    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-credential-socket\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704460 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.704283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-certs\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704460 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.704352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-socket\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704460 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.704396    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.704559 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.704490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-data\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.706001 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.705977    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-envoy\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.706231 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.706214    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-podinfo\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.712201 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.712176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5q9\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-kube-api-access-2q5q9\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.712332 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.712260    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-token\") pod \"router-gateway-1-openshift-default-758f598944-wtg76\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:58.818450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:58.818369    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:46:59.097328 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:46:59.097296    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"]
Sep 18 16:46:59.100026 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:46:59.099992    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11aa924_8149_44bd_86fd_c8f1a8d4d1df.slice/crio-0e2b7044bd7e10b22a375ab606674b740fb0d35bae3681df868fe8587ee54223 WatchSource:0}: Error finding container 0e2b7044bd7e10b22a375ab606674b740fb0d35bae3681df868fe8587ee54223: Status 404 returned error can't find the container with id 0e2b7044bd7e10b22a375ab606674b740fb0d35bae3681df868fe8587ee54223
Sep 18 16:47:00.111562 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:00.111520    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76" event={"ID":"c11aa924-8149-44bd-86fd-c8f1a8d4d1df","Type":"ContainerStarted","Data":"0e2b7044bd7e10b22a375ab606674b740fb0d35bae3681df868fe8587ee54223"}
Sep 18 16:47:01.937119 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:01.937076    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 16:47:01.937478 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:01.937153    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 16:47:01.937478 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:01.937182    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 16:47:02.125878 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:02.125824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76" event={"ID":"c11aa924-8149-44bd-86fd-c8f1a8d4d1df","Type":"ContainerStarted","Data":"f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb"}
Sep 18 16:47:02.151494 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:02.151422    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76" podStartSLOduration=1.316587058 podStartE2EDuration="4.151400916s" totalImagesPullingTime="2.834813858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:47:02.148451882 +0000 UTC m=+1855.923884062" watchObservedRunningTime="2026-09-18 16:47:02.151400916 +0000 UTC m=+1855.926833078"
Sep 18 16:47:02.818900 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:02.818859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:47:03.824578 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:03.824547    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:47:04.136619 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:04.136588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:47:04.137657 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:04.137631    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:47:20.179532 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:20.179421    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76" podUID="c11aa924-8149-44bd-86fd-c8f1a8d4d1df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:47:20.381065 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:20.381024    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"]
Sep 18 16:47:20.381335 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:20.381289    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76" podUID="c11aa924-8149-44bd-86fd-c8f1a8d4d1df" containerName="istio-proxy" containerID="cri-o://f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb" gracePeriod=30
Sep 18 16:47:21.735189 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.735161    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:47:21.817307 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-socket\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817329    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istiod-ca-cert\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817384    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-podinfo\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817470    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-credential-socket\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817502    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-envoy\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817533    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-certs\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817564    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2q5q9\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-kube-api-access-2q5q9\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:21.817735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-token\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.817735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817618    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-data\") pod \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\" (UID: \"c11aa924-8149-44bd-86fd-c8f1a8d4d1df\") "
Sep 18 16:47:21.818003 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817801    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:21.818003 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817828    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:21.818003 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817851    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-socket\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.818003 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.817858    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:47:21.818210 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.818082    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-data" (OuterVolumeSpecName: "istio-data") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:21.820593 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.820550    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:47:21.820735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.820640    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-kube-api-access-2q5q9" (OuterVolumeSpecName: "kube-api-access-2q5q9") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "kube-api-access-2q5q9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:21.820807 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.820725    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-token" (OuterVolumeSpecName: "istio-token") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:47:21.820807 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.820766    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c11aa924-8149-44bd-86fd-c8f1a8d4d1df" (UID: "c11aa924-8149-44bd-86fd-c8f1a8d4d1df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 16:47:21.918312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918276    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-token\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.918312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918309    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-data\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.918312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918320    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istiod-ca-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.918595 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918331    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-podinfo\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.918595 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918342    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-credential-socket\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.918595 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918355    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-istio-envoy\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.918595 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918365    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-workload-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:21.918595 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:21.918375    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q5q9\" (UniqueName: \"kubernetes.io/projected/c11aa924-8149-44bd-86fd-c8f1a8d4d1df-kube-api-access-2q5q9\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:47:22.250107 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.250068    2562 generic.go:356] "Generic (PLEG): container finished" podID="c11aa924-8149-44bd-86fd-c8f1a8d4d1df" containerID="f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb" exitCode=0
Sep 18 16:47:22.250350 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.250124    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"
Sep 18 16:47:22.250350 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.250157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76" event={"ID":"c11aa924-8149-44bd-86fd-c8f1a8d4d1df","Type":"ContainerDied","Data":"f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb"}
Sep 18 16:47:22.250350 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.250261    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76" event={"ID":"c11aa924-8149-44bd-86fd-c8f1a8d4d1df","Type":"ContainerDied","Data":"0e2b7044bd7e10b22a375ab606674b740fb0d35bae3681df868fe8587ee54223"}
Sep 18 16:47:22.250350 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.250288    2562 scope.go:117] "RemoveContainer" containerID="f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb"
Sep 18 16:47:22.259820 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.259802    2562 scope.go:117] "RemoveContainer" containerID="f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb"
Sep 18 16:47:22.260129 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:47:22.260110    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb\": container with ID starting with f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb not found: ID does not exist" containerID="f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb"
Sep 18 16:47:22.260172 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.260139    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb"} err="failed to get container status \"f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb\": rpc error: code = NotFound desc = could not find container \"f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb\": container with ID starting with f9891f8118f80512a99542753e4a2304d057e9f9fb1d9aa7da5ef2cd689cd4fb not found: ID does not exist"
Sep 18 16:47:22.272577 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.272541    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"]
Sep 18 16:47:22.278747 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.278716    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-758f598944-wtg76"]
Sep 18 16:47:22.334669 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:22.334635    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c11aa924-8149-44bd-86fd-c8f1a8d4d1df" path="/var/lib/kubelet/pods/c11aa924-8149-44bd-86fd-c8f1a8d4d1df/volumes"
Sep 18 16:47:43.883068 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.883025    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"]
Sep 18 16:47:43.883672 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.883589    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11aa924-8149-44bd-86fd-c8f1a8d4d1df" containerName="istio-proxy"
Sep 18 16:47:43.883672 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.883613    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11aa924-8149-44bd-86fd-c8f1a8d4d1df" containerName="istio-proxy"
Sep 18 16:47:43.883792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.883735    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11aa924-8149-44bd-86fd-c8f1a8d4d1df" containerName="istio-proxy"
Sep 18 16:47:43.888193 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.888165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.891102 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.891063    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 18 16:47:43.891296 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.891072    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-lwjsc\""
Sep 18 16:47:43.891296 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.891212    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:47:43.891971 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.891949    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 18 16:47:43.892149 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.891976    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 18 16:47:43.892149 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.891979    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 18 16:47:43.900189 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.900159    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"]
Sep 18 16:47:43.914792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.914757    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-dshm\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.914792 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.914798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-model-cache\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.915008 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.914824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tmp-dir\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.915008 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.914922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-cabundle-cert\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.915008 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.914950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv9sf\" (UniqueName: \"kubernetes.io/projected/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kube-api-access-dv9sf\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.915008 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.914979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tls-certs\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.915134 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.915026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kserve-provision-location\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:43.915134 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:43.915058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-home\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kserve-provision-location\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-home\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-dshm\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-model-cache\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016168    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tmp-dir\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-cabundle-cert\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv9sf\" (UniqueName: \"kubernetes.io/projected/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kube-api-access-dv9sf\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016312 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016289    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tls-certs\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016661 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kserve-provision-location\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016661 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016630    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-home\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-model-cache\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.016835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016824    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tmp-dir\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.017000 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.016973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-cabundle-cert\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.019166 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.019140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-dshm\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.019310 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.019162    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tls-certs\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.028271 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.028226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv9sf\" (UniqueName: \"kubernetes.io/projected/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kube-api-access-dv9sf\") pod \"auth-enabled-test-kserve-84b574c694-r5mxz\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.199962 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.199863    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:47:44.482423 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:44.482330    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"]
Sep 18 16:47:44.485664 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:47:44.485635    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9481e3bc_e6cd_4474_b0cc_8d0b42eab446.slice/crio-9ff91eefd556eb083d4df9f98d4b8389fd557971cc14d70650e52937685d1fdb WatchSource:0}: Error finding container 9ff91eefd556eb083d4df9f98d4b8389fd557971cc14d70650e52937685d1fdb: Status 404 returned error can't find the container with id 9ff91eefd556eb083d4df9f98d4b8389fd557971cc14d70650e52937685d1fdb
Sep 18 16:47:45.378537 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:45.378500    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" event={"ID":"9481e3bc-e6cd-4474-b0cc-8d0b42eab446","Type":"ContainerStarted","Data":"9ff91eefd556eb083d4df9f98d4b8389fd557971cc14d70650e52937685d1fdb"}
Sep 18 16:47:48.398878 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:48.398825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" event={"ID":"9481e3bc-e6cd-4474-b0cc-8d0b42eab446","Type":"ContainerStarted","Data":"7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8"}
Sep 18 16:47:50.412479 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:50.412445    2562 generic.go:356] "Generic (PLEG): container finished" podID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerID="7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8" exitCode=0
Sep 18 16:47:50.412967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:47:50.412518    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" event={"ID":"9481e3bc-e6cd-4474-b0cc-8d0b42eab446","Type":"ContainerDied","Data":"7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8"}
Sep 18 16:48:19.616890 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:19.616843    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" event={"ID":"9481e3bc-e6cd-4474-b0cc-8d0b42eab446","Type":"ContainerStarted","Data":"4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e"}
Sep 18 16:48:19.661892 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:19.661834    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podStartSLOduration=2.229647529 podStartE2EDuration="36.661814529s" totalImagesPullingTime="32.193780019s" totalInitContainerRuntime="2.238386981s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:47:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:48:19.65909351 +0000 UTC m=+1933.434525693" watchObservedRunningTime="2026-09-18 16:48:19.661814529 +0000 UTC m=+1933.437246693"
Sep 18 16:48:20.200633 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:20.200582    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:48:24.201133 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:24.201090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:48:24.202990 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:24.202955    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:48:34.201424 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:34.201369    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:48:44.200617 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:44.200564    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:48:54.201216 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:48:54.201163    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:49:04.201317 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:04.201265    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:49:14.200794 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:14.200737    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:49:15.792791 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.792744    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"]
Sep 18 16:49:15.796022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.795994    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.798981 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.798950    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 18 16:49:15.798981 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.798967    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:49:15.799185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.798991    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-bptcr\""
Sep 18 16:49:15.799185 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.798996    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 18 16:49:15.799948 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.799927    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 18 16:49:15.800053 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.799950    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 18 16:49:15.813250 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.813207    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"]
Sep 18 16:49:15.863539 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.863714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863559    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.863714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863586    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.863714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863616    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.863714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.863714 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863686    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6dmp\" (UniqueName: \"kubernetes.io/projected/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kube-api-access-f6dmp\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.863889 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863717    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.863889 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.863742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964403 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964602 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964433    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6dmp\" (UniqueName: \"kubernetes.io/projected/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kube-api-access-f6dmp\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964602 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964602 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964525    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964602 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964572    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.964875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964690    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.965033 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964903    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.965033 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.964940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-model-cache\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.965179 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.965156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-home\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.965367 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.965343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.965676 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.965653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.966967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.966940    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-dshm\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.967195 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.967178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tls-certs\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:15.973260 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:15.973219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6dmp\" (UniqueName: \"kubernetes.io/projected/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kube-api-access-f6dmp\") pod \"tokenizer-clean-path-test-kserve-554db9c97d-9cql4\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:16.107360 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:16.107262    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:16.392149 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:16.392108    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"]
Sep 18 16:49:16.392326 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:49:16.392118    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc25d0f_8b19_4c96_bca0_d8c8d93d14de.slice/crio-45c6a9bb8d5c8a7c3ff5294eddf3aef8044a95112edfff0228415032b9a1756d WatchSource:0}: Error finding container 45c6a9bb8d5c8a7c3ff5294eddf3aef8044a95112edfff0228415032b9a1756d: Status 404 returned error can't find the container with id 45c6a9bb8d5c8a7c3ff5294eddf3aef8044a95112edfff0228415032b9a1756d
Sep 18 16:49:16.956582 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:16.956535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" event={"ID":"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de","Type":"ContainerStarted","Data":"6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd"}
Sep 18 16:49:16.957059 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:16.956608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" event={"ID":"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de","Type":"ContainerStarted","Data":"45c6a9bb8d5c8a7c3ff5294eddf3aef8044a95112edfff0228415032b9a1756d"}
Sep 18 16:49:19.976175 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:19.976136    2562 generic.go:356] "Generic (PLEG): container finished" podID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerID="6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd" exitCode=0
Sep 18 16:49:19.976686 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:19.976212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" event={"ID":"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de","Type":"ContainerDied","Data":"6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd"}
Sep 18 16:49:20.983712 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:20.983667    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" event={"ID":"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de","Type":"ContainerStarted","Data":"5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85"}
Sep 18 16:49:21.010606 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:21.010535    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" podStartSLOduration=3.392253111 podStartE2EDuration="6.010506888s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.618253777s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:49:21.006778842 +0000 UTC m=+1994.782211018" watchObservedRunningTime="2026-09-18 16:49:21.010506888 +0000 UTC m=+1994.785939050"
Sep 18 16:49:21.118080 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:21.118024    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:24.200574 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:24.200521    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:49:26.108229 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:26.108178    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:26.121487 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:26.121458    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:27.033223 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:27.033192    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:34.200542 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:34.200494    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:49:44.200527 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:44.200473    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:49:53.663693 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:53.663593    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"]
Sep 18 16:49:53.675124 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:49:53.675079    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:49:53.675124 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:49:53.675124 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:49:53.675124 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85" cmd=["/bin/sleep","15"]
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:53.675145    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" message="" err=<
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:49:53.675158    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="main" containerID="cri-o://5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85"
Sep 18 16:49:53.675491 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:53.675214    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="main" containerID="cri-o://5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85" gracePeriod=60
Sep 18 16:49:54.047110 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.047075    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:54.184387 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.184348    2562 generic.go:356] "Generic (PLEG): container finished" podID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerID="5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85" exitCode=0
Sep 18 16:49:54.184561 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.184419    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"
Sep 18 16:49:54.184561 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.184429    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" event={"ID":"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de","Type":"ContainerDied","Data":"5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85"}
Sep 18 16:49:54.184561 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.184488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" event={"ID":"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de","Type":"ContainerDied","Data":"45c6a9bb8d5c8a7c3ff5294eddf3aef8044a95112edfff0228415032b9a1756d"}
Sep 18 16:49:54.184561 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.184518    2562 scope.go:117] "RemoveContainer" containerID="5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85"
Sep 18 16:49:54.195052 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.195024    2562 scope.go:117] "RemoveContainer" containerID="6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd"
Sep 18 16:49:54.200743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.200709    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:49:54.206105 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.206080    2562 scope.go:117] "RemoveContainer" containerID="5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85"
Sep 18 16:49:54.206560 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:49:54.206533    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85\": container with ID starting with 5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85 not found: ID does not exist" containerID="5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85"
Sep 18 16:49:54.206649 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.206568    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85"} err="failed to get container status \"5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85\": rpc error: code = NotFound desc = could not find container \"5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85\": container with ID starting with 5c117bf6eb10facb22c3c54548a713e5685eacb546cf16bc9d176928a6e21e85 not found: ID does not exist"
Sep 18 16:49:54.206649 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.206587    2562 scope.go:117] "RemoveContainer" containerID="6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd"
Sep 18 16:49:54.206917 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:49:54.206893    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd\": container with ID starting with 6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd not found: ID does not exist" containerID="6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd"
Sep 18 16:49:54.206961 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.206927    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd"} err="failed to get container status \"6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd\": rpc error: code = NotFound desc = could not find container \"6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd\": container with ID starting with 6e467291587f7981cfbb7e621fb65b3c209474414c61c0c57c406c1e5dabf0cd not found: ID does not exist"
Sep 18 16:49:54.225735 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225702    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-cabundle-cert\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.225926 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225786    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tmp-dir\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.225926 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225810    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-home\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.225926 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225827    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-model-cache\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.225926 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225856    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-dshm\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.225926 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tls-certs\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.226191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225940    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6dmp\" (UniqueName: \"kubernetes.io/projected/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kube-api-access-f6dmp\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.226191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.225986    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kserve-provision-location\") pod \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\" (UID: \"4cc25d0f-8b19-4c96-bca0-d8c8d93d14de\") "
Sep 18 16:49:54.226191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226052    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.226191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226069    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-home" (OuterVolumeSpecName: "home") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.226191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226142    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:49:54.226191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226148    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.226463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226313    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.226463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226357    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.226463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226371    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.226463 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.226383    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.228772 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.228742    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:49:54.228772 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.228750    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-dshm" (OuterVolumeSpecName: "dshm") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.228985 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.228777    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kube-api-access-f6dmp" (OuterVolumeSpecName: "kube-api-access-f6dmp") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "kube-api-access-f6dmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:49:54.245632 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.245578    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" (UID: "4cc25d0f-8b19-4c96-bca0-d8c8d93d14de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:49:54.327724 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.327687    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.327724 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.327726    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.327938 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.327743    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6dmp\" (UniqueName: \"kubernetes.io/projected/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kube-api-access-f6dmp\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.327938 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.327761    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:49:54.502454 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.502347    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"]
Sep 18 16:49:54.507580 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:54.507542    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4"]
Sep 18 16:49:55.023063 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:55.023017    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-554db9c97d-9cql4" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded"
Sep 18 16:49:56.334563 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:49:56.334526    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" path="/var/lib/kubelet/pods/4cc25d0f-8b19-4c96-bca0-d8c8d93d14de/volumes"
Sep 18 16:50:04.200944 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:04.200894    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:50:14.200393 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:14.200347    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:50:24.212558 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:24.212522    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:50:24.221530 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:24.221497    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:50:29.974885 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:29.974844    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"]
Sep 18 16:50:45.000933 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.000872    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" containerID="cri-o://4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e" gracePeriod=45
Sep 18 16:50:45.213169 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.213118    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Sep 18 16:50:45.755081 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.755053    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:50:45.892243 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892187    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-cabundle-cert\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892428 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892409    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dv9sf\" (UniqueName: \"kubernetes.io/projected/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kube-api-access-dv9sf\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892493 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892473    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tls-certs\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892529 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892512    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kserve-provision-location\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892573 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892561    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-model-cache\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892608 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892593    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-dshm\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892658 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892636    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:50:45.892710 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892660    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tmp-dir\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892815 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892796    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-home\") pod \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\" (UID: \"9481e3bc-e6cd-4474-b0cc-8d0b42eab446\") "
Sep 18 16:50:45.892969 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.892895    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-model-cache" (OuterVolumeSpecName: "model-cache") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:50:45.893153 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.893125    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:45.893280 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.893157    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:45.893525 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.893498    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-home" (OuterVolumeSpecName: "home") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:50:45.895418 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.895383    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-dshm" (OuterVolumeSpecName: "dshm") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:50:45.895503 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.895415    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kube-api-access-dv9sf" (OuterVolumeSpecName: "kube-api-access-dv9sf") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "kube-api-access-dv9sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:50:45.895503 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.895468    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:50:45.906078 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.906039    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:50:45.913914 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.913868    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9481e3bc-e6cd-4474-b0cc-8d0b42eab446" (UID: "9481e3bc-e6cd-4474-b0cc-8d0b42eab446"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:50:45.994367 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.994324    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv9sf\" (UniqueName: \"kubernetes.io/projected/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kube-api-access-dv9sf\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:45.994367 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.994358    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:45.994367 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.994367    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:45.994367 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.994376    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:45.994633 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.994386    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:45.994633 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:45.994394    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9481e3bc-e6cd-4474-b0cc-8d0b42eab446-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:50:46.481939 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.481898    2562 generic.go:356] "Generic (PLEG): container finished" podID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerID="4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e" exitCode=0
Sep 18 16:50:46.482401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.481961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" event={"ID":"9481e3bc-e6cd-4474-b0cc-8d0b42eab446","Type":"ContainerDied","Data":"4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e"}
Sep 18 16:50:46.482401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.482008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz" event={"ID":"9481e3bc-e6cd-4474-b0cc-8d0b42eab446","Type":"ContainerDied","Data":"9ff91eefd556eb083d4df9f98d4b8389fd557971cc14d70650e52937685d1fdb"}
Sep 18 16:50:46.482401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.482034    2562 scope.go:117] "RemoveContainer" containerID="4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e"
Sep 18 16:50:46.482401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.481963    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"
Sep 18 16:50:46.491762 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.491732    2562 scope.go:117] "RemoveContainer" containerID="7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8"
Sep 18 16:50:46.501515 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.501489    2562 scope.go:117] "RemoveContainer" containerID="4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e"
Sep 18 16:50:46.501905 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:50:46.501881    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e\": container with ID starting with 4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e not found: ID does not exist" containerID="4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e"
Sep 18 16:50:46.501995 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.501918    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e"} err="failed to get container status \"4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e\": rpc error: code = NotFound desc = could not find container \"4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e\": container with ID starting with 4ee312c49cb0392fa74e9094d6f57283228c594536d0cfb5e2e38414fe73f97e not found: ID does not exist"
Sep 18 16:50:46.501995 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.501946    2562 scope.go:117] "RemoveContainer" containerID="7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8"
Sep 18 16:50:46.502313 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:50:46.502291    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8\": container with ID starting with 7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8 not found: ID does not exist" containerID="7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8"
Sep 18 16:50:46.502375 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.502322    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8"} err="failed to get container status \"7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8\": rpc error: code = NotFound desc = could not find container \"7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8\": container with ID starting with 7ca9534c5ece21e63f50bda4043d20821e95151c28238e5e4b718689607b31b8 not found: ID does not exist"
Sep 18 16:50:46.502667 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.502641    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"]
Sep 18 16:50:46.506159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:46.506134    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-84b574c694-r5mxz"]
Sep 18 16:50:48.334042 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:48.334007    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" path="/var/lib/kubelet/pods/9481e3bc-e6cd-4474-b0cc-8d0b42eab446/volumes"
Sep 18 16:50:50.334919 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.334887    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"]
Sep 18 16:50:50.335322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335278    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main"
Sep 18 16:50:50.335322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335291    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main"
Sep 18 16:50:50.335322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335313    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="storage-initializer"
Sep 18 16:50:50.335322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335320    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="storage-initializer"
Sep 18 16:50:50.335469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335327    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="main"
Sep 18 16:50:50.335469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335335    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="main"
Sep 18 16:50:50.335469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335345    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="storage-initializer"
Sep 18 16:50:50.335469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335350    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="storage-initializer"
Sep 18 16:50:50.335469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335414    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9481e3bc-e6cd-4474-b0cc-8d0b42eab446" containerName="main"
Sep 18 16:50:50.335469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.335421    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc25d0f-8b19-4c96-bca0-d8c8d93d14de" containerName="main"
Sep 18 16:50:50.338820 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.338794    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.341536 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.341511    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 18 16:50:50.341798 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.341766    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-xcr42\""
Sep 18 16:50:50.341894 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.341876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 18 16:50:50.341954 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.341911    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 18 16:50:50.342760 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.342743    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 18 16:50:50.342856 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.342781    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:50:50.356870 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.356836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"]
Sep 18 16:50:50.432781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.432740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.432966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.432791    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.432966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.432850    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.432966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.432873    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.432966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.432922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.432966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.432945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.433128 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.433020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bc5bba5-9f54-4f33-b919-1c0179d6808e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.433128 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.433111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-986xv\" (UniqueName: \"kubernetes.io/projected/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kube-api-access-986xv\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.534869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-986xv\" (UniqueName: \"kubernetes.io/projected/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kube-api-access-986xv\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.534959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.535002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.535052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.535086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.535142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.535177    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.535222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bc5bba5-9f54-4f33-b919-1c0179d6808e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.536186    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bc5bba5-9f54-4f33-b919-1c0179d6808e-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.536921    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-model-cache\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.538215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.537176    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.539744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.539709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.539902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.539848    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-home\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.542854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.542100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tls-certs\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.542854 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.542732    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-dshm\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.560843 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.560808    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-986xv\" (UniqueName: \"kubernetes.io/projected/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kube-api-access-986xv\") pod \"tokenizer-migration-path-test-kserve-74855cb8b7-8btnh\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.649440 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.649393    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:50:50.931452 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:50.931423    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"]
Sep 18 16:50:50.933540 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:50:50.933511    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc5bba5_9f54_4f33_b919_1c0179d6808e.slice/crio-8c421cd481bb73327fc6ea0c8a2498e1be4b553cbda1fa64e3d7270038028f91 WatchSource:0}: Error finding container 8c421cd481bb73327fc6ea0c8a2498e1be4b553cbda1fa64e3d7270038028f91: Status 404 returned error can't find the container with id 8c421cd481bb73327fc6ea0c8a2498e1be4b553cbda1fa64e3d7270038028f91
Sep 18 16:50:51.513374 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:51.513334    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" event={"ID":"0bc5bba5-9f54-4f33-b919-1c0179d6808e","Type":"ContainerStarted","Data":"fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1"}
Sep 18 16:50:51.513884 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:51.513390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" event={"ID":"0bc5bba5-9f54-4f33-b919-1c0179d6808e","Type":"ContainerStarted","Data":"8c421cd481bb73327fc6ea0c8a2498e1be4b553cbda1fa64e3d7270038028f91"}
Sep 18 16:50:53.525905 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:53.525863    2562 generic.go:356] "Generic (PLEG): container finished" podID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerID="fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1" exitCode=0
Sep 18 16:50:53.526341 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:53.525933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" event={"ID":"0bc5bba5-9f54-4f33-b919-1c0179d6808e","Type":"ContainerDied","Data":"fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1"}
Sep 18 16:50:54.533629 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:54.533594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" event={"ID":"0bc5bba5-9f54-4f33-b919-1c0179d6808e","Type":"ContainerStarted","Data":"fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f"}
Sep 18 16:50:54.555744 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:54.555689    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" podStartSLOduration=2.194332686 podStartE2EDuration="4.555671989s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.361339303s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:50:54.553175538 +0000 UTC m=+2088.328607756" watchObservedRunningTime="2026-09-18 16:50:54.555671989 +0000 UTC m=+2088.331104175"
Sep 18 16:50:54.649983 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:50:54.649951    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:51:00.650315 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:00.650280    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:51:00.663210 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:00.663181    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:51:01.585226 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:01.585191    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:51:34.298214 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.298177    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"]
Sep 18 16:51:34.301756 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.301735    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.309821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.305035    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:51:34.309821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.305421    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 18 16:51:34.309821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.305434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-prdmf\""
Sep 18 16:51:34.309821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.305469    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 18 16:51:34.309821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.305039    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 18 16:51:34.309821 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.306182    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 18 16:51:34.317217 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.317191    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"]
Sep 18 16:51:34.413808 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.413766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.413808 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.413809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.414074 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.413872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-home\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.414074 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.413924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-dshm\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.414074 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.413976    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tmp-dir\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.414074 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.414005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-model-cache\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.414074 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.414049    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tls-certs\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.414074 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.414069    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfd66\" (UniqueName: \"kubernetes.io/projected/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kube-api-access-sfd66\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.514761 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514717    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfd66\" (UniqueName: \"kubernetes.io/projected/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kube-api-access-sfd66\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.514982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514803    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.514982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514841    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.514982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514872    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-home\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.514982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514900    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-dshm\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.514982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tmp-dir\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.514982 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514968    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-model-cache\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.515344 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.514999    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tls-certs\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.515407 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.515343    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.515407 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.515369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-home\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.515554 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.515429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tmp-dir\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.515554 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.515441    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-model-cache\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.515707 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.515681    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-cabundle-cert\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.517684 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.517621    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-dshm\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.518152 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.518131    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tls-certs\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.523813 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.523785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfd66\" (UniqueName: \"kubernetes.io/projected/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kube-api-access-sfd66\") pod \"auth-invalid-token-test-kserve-8769cf786-krdx9\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.620868 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.620830    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:34.908729 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:34.908696    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"]
Sep 18 16:51:34.911576 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:51:34.911538    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd6a27c5_1808_4d90_9cd8_99d29d60ef23.slice/crio-55f0dee96bcfe5242a96dbf0d63903e76bce60289075111617077b4aecec0598 WatchSource:0}: Error finding container 55f0dee96bcfe5242a96dbf0d63903e76bce60289075111617077b4aecec0598: Status 404 returned error can't find the container with id 55f0dee96bcfe5242a96dbf0d63903e76bce60289075111617077b4aecec0598
Sep 18 16:51:35.778016 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:35.777978    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" event={"ID":"cd6a27c5-1808-4d90-9cd8-99d29d60ef23","Type":"ContainerStarted","Data":"433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e"}
Sep 18 16:51:35.778405 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:35.778034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" event={"ID":"cd6a27c5-1808-4d90-9cd8-99d29d60ef23","Type":"ContainerStarted","Data":"55f0dee96bcfe5242a96dbf0d63903e76bce60289075111617077b4aecec0598"}
Sep 18 16:51:37.795030 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:37.794997    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerID="433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e" exitCode=0
Sep 18 16:51:37.795453 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:37.795069    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" event={"ID":"cd6a27c5-1808-4d90-9cd8-99d29d60ef23","Type":"ContainerDied","Data":"433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e"}
Sep 18 16:51:38.803081 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:38.803046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" event={"ID":"cd6a27c5-1808-4d90-9cd8-99d29d60ef23","Type":"ContainerStarted","Data":"3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb"}
Sep 18 16:51:38.828304 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:38.828216    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podStartSLOduration=2.498852732 podStartE2EDuration="4.828198744s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.329346012s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:51:38.824714918 +0000 UTC m=+2132.600147080" watchObservedRunningTime="2026-09-18 16:51:38.828198744 +0000 UTC m=+2132.603630906"
Sep 18 16:51:39.621307 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:39.621265    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:43.864797 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:43.864759    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"]
Sep 18 16:51:43.876172 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:51:43.876133    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:51:43.876172 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:51:43.876172 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:51:43.876172 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f" cmd=["/bin/sleep","15"]
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:43.876188    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" message="" err=<
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:51:43.876204    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="main" containerID="cri-o://fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f"
Sep 18 16:51:43.876550 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:43.876257    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="main" containerID="cri-o://fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f" gracePeriod=60
Sep 18 16:51:44.621877 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.621838    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:51:44.623595 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.623556    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:51:44.647902 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.647873    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:51:44.812845 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.812748    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-model-cache\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.812845 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.812793    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-dshm\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.812845 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.812817    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-home\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.812845 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.812843    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tls-certs\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.813188 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.812874    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tmp-dir\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.813279 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.812955    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bc5bba5-9f54-4f33-b919-1c0179d6808e-cabundle-cert\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.813345 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813324    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kserve-provision-location\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.813401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813046    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-home" (OuterVolumeSpecName: "home") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.813401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813060    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-model-cache" (OuterVolumeSpecName: "model-cache") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.813401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813153    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.813401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813317    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc5bba5-9f54-4f33-b919-1c0179d6808e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:51:44.813401 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813386    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-986xv\" (UniqueName: \"kubernetes.io/projected/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kube-api-access-986xv\") pod \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\" (UID: \"0bc5bba5-9f54-4f33-b919-1c0179d6808e\") "
Sep 18 16:51:44.813683 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813663    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bc5bba5-9f54-4f33-b919-1c0179d6808e-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.813741 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813689    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.813741 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813703    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.813741 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.813716    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.815689 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.815656    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:51:44.815834 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.815659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-dshm" (OuterVolumeSpecName: "dshm") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.815910 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.815854    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kube-api-access-986xv" (OuterVolumeSpecName: "kube-api-access-986xv") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "kube-api-access-986xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:51:44.837724 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.837676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bc5bba5-9f54-4f33-b919-1c0179d6808e" (UID: "0bc5bba5-9f54-4f33-b919-1c0179d6808e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:51:44.840621 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.840588    2562 generic.go:356] "Generic (PLEG): container finished" podID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerID="fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f" exitCode=0
Sep 18 16:51:44.840769 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.840639    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"
Sep 18 16:51:44.840769 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.840665    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" event={"ID":"0bc5bba5-9f54-4f33-b919-1c0179d6808e","Type":"ContainerDied","Data":"fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f"}
Sep 18 16:51:44.840769 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.840722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" event={"ID":"0bc5bba5-9f54-4f33-b919-1c0179d6808e","Type":"ContainerDied","Data":"8c421cd481bb73327fc6ea0c8a2498e1be4b553cbda1fa64e3d7270038028f91"}
Sep 18 16:51:44.840769 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.840745    2562 scope.go:117] "RemoveContainer" containerID="fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f"
Sep 18 16:51:44.853346 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.853315    2562 scope.go:117] "RemoveContainer" containerID="fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1"
Sep 18 16:51:44.862754 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.862719    2562 scope.go:117] "RemoveContainer" containerID="fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f"
Sep 18 16:51:44.863176 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:51:44.863155    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f\": container with ID starting with fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f not found: ID does not exist" containerID="fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f"
Sep 18 16:51:44.863300 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.863185    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f"} err="failed to get container status \"fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f\": rpc error: code = NotFound desc = could not find container \"fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f\": container with ID starting with fda7826deaf03ec41d9bf9c666a09861b88c6417c397843d88646644ec155b1f not found: ID does not exist"
Sep 18 16:51:44.863300 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.863206    2562 scope.go:117] "RemoveContainer" containerID="fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1"
Sep 18 16:51:44.863546 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:51:44.863525    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1\": container with ID starting with fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1 not found: ID does not exist" containerID="fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1"
Sep 18 16:51:44.863595 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.863553    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1"} err="failed to get container status \"fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1\": rpc error: code = NotFound desc = could not find container \"fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1\": container with ID starting with fbc8340528a3e93fdb9978d4dc4188b6492a0da78c9aa3f50588a0d8c512eba1 not found: ID does not exist"
Sep 18 16:51:44.866632 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.866607    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"]
Sep 18 16:51:44.871127 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.871102    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh"]
Sep 18 16:51:44.914725 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.914687    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0bc5bba5-9f54-4f33-b919-1c0179d6808e-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.914725 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.914720    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.914725 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.914735    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-986xv\" (UniqueName: \"kubernetes.io/projected/0bc5bba5-9f54-4f33-b919-1c0179d6808e-kube-api-access-986xv\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:44.914991 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:44.914745    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0bc5bba5-9f54-4f33-b919-1c0179d6808e-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:51:45.574130 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:45.574084    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-74855cb8b7-8btnh" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 16:51:46.335438 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:46.335402    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" path="/var/lib/kubelet/pods/0bc5bba5-9f54-4f33-b919-1c0179d6808e/volumes"
Sep 18 16:51:54.621879 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:51:54.621829    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:52:04.621699 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:04.621652    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:52:14.622170 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:14.622128    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:52:24.621712 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:24.621655    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:52:34.621665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:34.621620    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:52:43.284097 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.284057    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"]
Sep 18 16:52:43.284650 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.284507    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="storage-initializer"
Sep 18 16:52:43.284650 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.284525    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="storage-initializer"
Sep 18 16:52:43.284650 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.284543    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="main"
Sep 18 16:52:43.284650 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.284549    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="main"
Sep 18 16:52:43.284650 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.284618    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bc5bba5-9f54-4f33-b919-1c0179d6808e" containerName="main"
Sep 18 16:52:43.287907 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.287883    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.290727 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.290693    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 18 16:52:43.290888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.290733    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 18 16:52:43.290888 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.290693    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:52:43.291778 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.291752    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 18 16:52:43.291911 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.291793    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 18 16:52:43.291911 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.291762    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-h9l5s\""
Sep 18 16:52:43.300037 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.300012    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"]
Sep 18 16:52:43.345983 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.345947    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.345983 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.345986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.346246 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.346072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.346246 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.346138    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.346246 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.346195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.346384 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.346291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.346384 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.346331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrct7\" (UniqueName: \"kubernetes.io/projected/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kube-api-access-hrct7\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.346384 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.346365    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.447413 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.447372    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.447613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.447437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.447613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.447475    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrct7\" (UniqueName: \"kubernetes.io/projected/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kube-api-access-hrct7\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.447613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.447505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.447613 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.447540    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.447967 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.447940    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.448048 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.448027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.448104 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.447947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-home\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.448159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.448062    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.448211 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.448191    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-model-cache\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.448296 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.448271    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.448467 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.448446    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.448659 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.448638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.450026 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.449995    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-dshm\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.450212 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.450192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tls-certs\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.456039 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.456005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrct7\" (UniqueName: \"kubernetes.io/projected/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kube-api-access-hrct7\") pod \"tokenizer-template-args-test-kserve-7698f8bd97-2l4ct\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.600157 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.600066    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:43.885485 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:43.885458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"]
Sep 18 16:52:43.888094 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:52:43.888065    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63ebbd8_0b5d_43f5_b0b3_140347a251ba.slice/crio-28ae8520c63c6b539e72bdaf404016cd917f2a280c0660918ebc88aa95c841da WatchSource:0}: Error finding container 28ae8520c63c6b539e72bdaf404016cd917f2a280c0660918ebc88aa95c841da: Status 404 returned error can't find the container with id 28ae8520c63c6b539e72bdaf404016cd917f2a280c0660918ebc88aa95c841da
Sep 18 16:52:44.184637 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:44.184534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" event={"ID":"e63ebbd8-0b5d-43f5-b0b3-140347a251ba","Type":"ContainerStarted","Data":"89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9"}
Sep 18 16:52:44.184637 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:44.184594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" event={"ID":"e63ebbd8-0b5d-43f5-b0b3-140347a251ba","Type":"ContainerStarted","Data":"28ae8520c63c6b539e72bdaf404016cd917f2a280c0660918ebc88aa95c841da"}
Sep 18 16:52:44.621668 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:44.621622    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:52:47.205018 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:47.204979    2562 generic.go:356] "Generic (PLEG): container finished" podID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerID="89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9" exitCode=0
Sep 18 16:52:47.205388 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:47.205060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" event={"ID":"e63ebbd8-0b5d-43f5-b0b3-140347a251ba","Type":"ContainerDied","Data":"89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9"}
Sep 18 16:52:48.213307 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:48.213270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" event={"ID":"e63ebbd8-0b5d-43f5-b0b3-140347a251ba","Type":"ContainerStarted","Data":"e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef"}
Sep 18 16:52:48.235835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:48.235766    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" podStartSLOduration=2.753335447 podStartE2EDuration="5.235747541s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.482412094s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:52:48.234493452 +0000 UTC m=+2202.009925665" watchObservedRunningTime="2026-09-18 16:52:48.235747541 +0000 UTC m=+2202.011179703"
Sep 18 16:52:48.601293 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:48.601186    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:53.600261 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:53.600190    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:53.613136 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:53.613097    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:54.260949 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:54.260905    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:52:54.621444 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:52:54.621407    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:53:04.621607 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:04.621560    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:53:14.622121 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:14.622071    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:53:20.730496 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:20.730406    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"]
Sep 18 16:53:20.742986 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:53:20.742927    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:53:20.742986 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:53:20.742986 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:53:20.742986 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef" cmd=["/bin/sleep","15"]
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:20.743007    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" message="" err=<
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:53:20.743023    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerName="main" containerID="cri-o://e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef"
Sep 18 16:53:20.743338 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:20.743098    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerName="main" containerID="cri-o://e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef" gracePeriod=60
Sep 18 16:53:21.099508 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.099477    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:53:21.196251 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196210    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kserve-provision-location\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.196450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-dshm\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.196450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196294    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tmp-dir\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.196450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196315    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tls-certs\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.196450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196344    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrct7\" (UniqueName: \"kubernetes.io/projected/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kube-api-access-hrct7\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.196450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196383    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-model-cache\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.196450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196427    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-home\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.196765 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196600    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.196765 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196719    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.196864 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.196771    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-cabundle-cert\") pod \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\" (UID: \"e63ebbd8-0b5d-43f5-b0b3-140347a251ba\") "
Sep 18 16:53:21.197054 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.197000    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-home" (OuterVolumeSpecName: "home") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.197191 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.197166    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:53:21.197391 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.197365    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.197391 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.197392    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.197583 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.197406    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.197583 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.197420    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.198958 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.198929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:53:21.199119 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.199092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kube-api-access-hrct7" (OuterVolumeSpecName: "kube-api-access-hrct7") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "kube-api-access-hrct7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:53:21.199339 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.199308    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-dshm" (OuterVolumeSpecName: "dshm") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.224340 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.224295    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e63ebbd8-0b5d-43f5-b0b3-140347a251ba" (UID: "e63ebbd8-0b5d-43f5-b0b3-140347a251ba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:53:21.298404 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.298318    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrct7\" (UniqueName: \"kubernetes.io/projected/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kube-api-access-hrct7\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.298404 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.298355    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.298404 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.298370    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.298404 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.298384    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e63ebbd8-0b5d-43f5-b0b3-140347a251ba-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:53:21.415868 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.415820    2562 generic.go:356] "Generic (PLEG): container finished" podID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerID="e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef" exitCode=0
Sep 18 16:53:21.416062 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.415902    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"
Sep 18 16:53:21.416062 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.415896    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" event={"ID":"e63ebbd8-0b5d-43f5-b0b3-140347a251ba","Type":"ContainerDied","Data":"e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef"}
Sep 18 16:53:21.416062 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.416025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct" event={"ID":"e63ebbd8-0b5d-43f5-b0b3-140347a251ba","Type":"ContainerDied","Data":"28ae8520c63c6b539e72bdaf404016cd917f2a280c0660918ebc88aa95c841da"}
Sep 18 16:53:21.416062 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.416046    2562 scope.go:117] "RemoveContainer" containerID="e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef"
Sep 18 16:53:21.427261 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.427229    2562 scope.go:117] "RemoveContainer" containerID="89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9"
Sep 18 16:53:21.435915 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.435890    2562 scope.go:117] "RemoveContainer" containerID="e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef"
Sep 18 16:53:21.436269 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:53:21.436229    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef\": container with ID starting with e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef not found: ID does not exist" containerID="e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef"
Sep 18 16:53:21.436344 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.436278    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef"} err="failed to get container status \"e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef\": rpc error: code = NotFound desc = could not find container \"e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef\": container with ID starting with e104f90f0de94eb327321705b277cd470debde2cb78ad939bbcd9e5710792aef not found: ID does not exist"
Sep 18 16:53:21.436344 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.436296    2562 scope.go:117] "RemoveContainer" containerID="89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9"
Sep 18 16:53:21.436592 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:53:21.436573    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9\": container with ID starting with 89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9 not found: ID does not exist" containerID="89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9"
Sep 18 16:53:21.436638 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.436599    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9"} err="failed to get container status \"89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9\": rpc error: code = NotFound desc = could not find container \"89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9\": container with ID starting with 89040a756a282f4c5d5de7f1a1d5854cacf0c5269fc15d0cc51d8dd0ba9b78e9 not found: ID does not exist"
Sep 18 16:53:21.442022 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.441993    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"]
Sep 18 16:53:21.450973 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:21.450942    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7698f8bd97-2l4ct"]
Sep 18 16:53:22.334427 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:22.334396    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" path="/var/lib/kubelet/pods/e63ebbd8-0b5d-43f5-b0b3-140347a251ba/volumes"
Sep 18 16:53:24.621660 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:24.621617    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:53:34.621824 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:34.621772    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:53:44.632172 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:44.632139    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:53:44.641076 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:44.641044    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:53:46.059666 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:53:46.059628    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"]
Sep 18 16:54:01.578044 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:01.578000    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" containerID="cri-o://3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb" gracePeriod=45
Sep 18 16:54:01.632891 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:01.632852    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": dial tcp 10.134.0.44:8000: connect: connection refused"
Sep 18 16:54:02.336633 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.336602    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:54:02.459086 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.458979    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-dshm\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459086 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459059    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfd66\" (UniqueName: \"kubernetes.io/projected/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kube-api-access-sfd66\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459107    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tmp-dir\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459161    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kserve-provision-location\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459194    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-home\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-model-cache\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459286    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-cabundle-cert\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459377 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459311    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tls-certs\") pod \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\" (UID: \"cd6a27c5-1808-4d90-9cd8-99d29d60ef23\") "
Sep 18 16:54:02.459679 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459618    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:02.459985 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.459953    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-home" (OuterVolumeSpecName: "home") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:02.460434 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.460406    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:54:02.461624 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.461596    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-dshm" (OuterVolumeSpecName: "dshm") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:02.461743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.461621    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kube-api-access-sfd66" (OuterVolumeSpecName: "kube-api-access-sfd66") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "kube-api-access-sfd66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:54:02.461743 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.461719    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:54:02.478332 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.478288    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:02.489777 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.489738    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd6a27c5-1808-4d90-9cd8-99d29d60ef23" (UID: "cd6a27c5-1808-4d90-9cd8-99d29d60ef23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:54:02.560546 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560504    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.560546 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560537    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfd66\" (UniqueName: \"kubernetes.io/projected/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kube-api-access-sfd66\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.560546 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560548    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.560824 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560558    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.560824 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560566    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.560824 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560575    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.560824 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560583    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.560824 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.560590    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd6a27c5-1808-4d90-9cd8-99d29d60ef23-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:54:02.644806 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.644762    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerID="3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb" exitCode=0
Sep 18 16:54:02.645218 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.644819    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"
Sep 18 16:54:02.645218 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.644837    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" event={"ID":"cd6a27c5-1808-4d90-9cd8-99d29d60ef23","Type":"ContainerDied","Data":"3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb"}
Sep 18 16:54:02.645218 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.644902    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9" event={"ID":"cd6a27c5-1808-4d90-9cd8-99d29d60ef23","Type":"ContainerDied","Data":"55f0dee96bcfe5242a96dbf0d63903e76bce60289075111617077b4aecec0598"}
Sep 18 16:54:02.645218 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.644926    2562 scope.go:117] "RemoveContainer" containerID="3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb"
Sep 18 16:54:02.655158 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.655137    2562 scope.go:117] "RemoveContainer" containerID="433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e"
Sep 18 16:54:02.664153 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.664125    2562 scope.go:117] "RemoveContainer" containerID="3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb"
Sep 18 16:54:02.664740 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:54:02.664711    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb\": container with ID starting with 3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb not found: ID does not exist" containerID="3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb"
Sep 18 16:54:02.664845 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.664753    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb"} err="failed to get container status \"3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb\": rpc error: code = NotFound desc = could not find container \"3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb\": container with ID starting with 3572846b5e109a0f5fe97a76d028ea39a6ea253e37480bdd033707cb76b33deb not found: ID does not exist"
Sep 18 16:54:02.664845 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.664780    2562 scope.go:117] "RemoveContainer" containerID="433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e"
Sep 18 16:54:02.665158 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:54:02.665099    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e\": container with ID starting with 433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e not found: ID does not exist" containerID="433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e"
Sep 18 16:54:02.665322 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.665161    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e"} err="failed to get container status \"433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e\": rpc error: code = NotFound desc = could not find container \"433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e\": container with ID starting with 433029b092df46bfb09b522704b3ca6715578e0ccf898effc3d8c90073e3c50e not found: ID does not exist"
Sep 18 16:54:02.667387 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.667362    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"]
Sep 18 16:54:02.670378 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:02.670353    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-8769cf786-krdx9"]
Sep 18 16:54:04.335435 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:04.335401    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" path="/var/lib/kubelet/pods/cd6a27c5-1808-4d90-9cd8-99d29d60ef23/volumes"
Sep 18 16:54:45.191325 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191286    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"]
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191682    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main"
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191694    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main"
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191709    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="storage-initializer"
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191715    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="storage-initializer"
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191721    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerName="main"
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191727    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerName="main"
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191740    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerName="storage-initializer"
Sep 18 16:54:45.191816 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191748    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerName="storage-initializer"
Sep 18 16:54:45.192084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191831    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e63ebbd8-0b5d-43f5-b0b3-140347a251ba" containerName="main"
Sep 18 16:54:45.192084 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.191843    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd6a27c5-1808-4d90-9cd8-99d29d60ef23" containerName="main"
Sep 18 16:54:45.195008 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.194990    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.197910 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.197880    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 18 16:54:45.198069 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.197886    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 18 16:54:45.198069 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.198002    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 18 16:54:45.198192 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.198150    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-hzp4w\""
Sep 18 16:54:45.198392 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.198372    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 18 16:54:45.198876 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.198857    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 16:54:45.209485 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.209447    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"]
Sep 18 16:54:45.213396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-home\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.213542 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tls-certs\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.213542 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213425    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-cabundle-cert\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.213542 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213531    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tmp-dir\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.213713 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kserve-provision-location\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.213713 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213639    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-model-cache\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.213713 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213683    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-dshm\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.213856 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.213742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9rwt\" (UniqueName: \"kubernetes.io/projected/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kube-api-access-m9rwt\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.314940 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.314907    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kserve-provision-location\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.314950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-model-cache\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.314977    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-dshm\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315008    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9rwt\" (UniqueName: \"kubernetes.io/projected/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kube-api-access-m9rwt\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-home\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315051    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tls-certs\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-cabundle-cert\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315146 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315104    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tmp-dir\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315551 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315378    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-model-cache\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315551 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315460    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kserve-provision-location\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315758 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tmp-dir\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.315866 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.315803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-home\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.316047 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.316020    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-cabundle-cert\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.317803 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.317776    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-dshm\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.318215 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.318150    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tls-certs\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.326643 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.326616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9rwt\" (UniqueName: \"kubernetes.io/projected/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kube-api-access-m9rwt\") pod \"auth-disabled-test-kserve-54f6b74bc-z77b4\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.506059 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.505958    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:45.791018 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.790990    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"]
Sep 18 16:54:45.793024 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:54:45.792988    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84d9ca22_43c5_4144_a0bb_0a5aa0e43a5c.slice/crio-56107cb1bddbd1406e3be5ad55f2401de59bb06998c2cea5bfb6b579b3db8947 WatchSource:0}: Error finding container 56107cb1bddbd1406e3be5ad55f2401de59bb06998c2cea5bfb6b579b3db8947: Status 404 returned error can't find the container with id 56107cb1bddbd1406e3be5ad55f2401de59bb06998c2cea5bfb6b579b3db8947
Sep 18 16:54:45.888655 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.888611    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" event={"ID":"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c","Type":"ContainerStarted","Data":"c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e"}
Sep 18 16:54:45.888828 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:45.888676    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" event={"ID":"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c","Type":"ContainerStarted","Data":"56107cb1bddbd1406e3be5ad55f2401de59bb06998c2cea5bfb6b579b3db8947"}
Sep 18 16:54:48.908392 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:48.908351    2562 generic.go:356] "Generic (PLEG): container finished" podID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerID="c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e" exitCode=0
Sep 18 16:54:48.908772 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:48.908415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" event={"ID":"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c","Type":"ContainerDied","Data":"c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e"}
Sep 18 16:54:49.916654 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:49.916614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" event={"ID":"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c","Type":"ContainerStarted","Data":"1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3"}
Sep 18 16:54:49.943093 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:49.943032    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podStartSLOduration=2.65371715 podStartE2EDuration="4.943013356s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.289296206s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:54:49.941383813 +0000 UTC m=+2323.716816001" watchObservedRunningTime="2026-09-18 16:54:49.943013356 +0000 UTC m=+2323.718445543"
Sep 18 16:54:50.506451 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:50.506408    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:55.507189 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:55.507144    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:54:55.508709 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:54:55.508679    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:55:05.506592 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:55:05.506538    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:55:15.506889 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:55:15.506843    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:55:25.507396 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:55:25.507345    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:55:35.506994 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:55:35.506940    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:55:45.506764 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:55:45.506697    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:55:55.507075 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:55:55.507018    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:56:05.507443 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:05.507400    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:56:15.507229 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:15.507165    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:56:25.507194 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:25.507148    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:56:35.506472 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:35.506427    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:56:45.507459 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:45.507402    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:56:48.679287 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.679219    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"]
Sep 18 16:56:48.686004 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.685977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.689658 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.689629    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 18 16:56:48.689658 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.689645    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-khwgj\""
Sep 18 16:56:48.689875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.689679    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 18 16:56:48.689875 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.689702    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 18 16:56:48.694334 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.694011    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"]
Sep 18 16:56:48.808300 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.808225    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf6xc\" (UniqueName: \"kubernetes.io/projected/e29dd90d-c910-42a0-bb83-293d428d4755-kube-api-access-vf6xc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.808495 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.808346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.808495 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.808388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.808495 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.808453    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dd90d-c910-42a0-bb83-293d428d4755-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.808633 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.808514    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.808633 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.808571    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.909731 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.909686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dd90d-c910-42a0-bb83-293d428d4755-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.909945 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.909785    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.909945 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.909851    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.909945 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.909889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf6xc\" (UniqueName: \"kubernetes.io/projected/e29dd90d-c910-42a0-bb83-293d428d4755-kube-api-access-vf6xc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.909945 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.909920    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.910190 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.909952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.910412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.910374    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.910524 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.910409    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.910623 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.910602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.912391 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.912370    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.912700 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.912679    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dd90d-c910-42a0-bb83-293d428d4755-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.918211 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.918184    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf6xc\" (UniqueName: \"kubernetes.io/projected/e29dd90d-c910-42a0-bb83-293d428d4755-kube-api-access-vf6xc\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:48.997867 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:48.997781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:49.277950 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:49.277918    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"]
Sep 18 16:56:49.280343 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:56:49.280302    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29dd90d_c910_42a0_bb83_293d428d4755.slice/crio-5bcc1e68e02cc00fb92d221602740758f42379c878600a7480da37721a6742c9 WatchSource:0}: Error finding container 5bcc1e68e02cc00fb92d221602740758f42379c878600a7480da37721a6742c9: Status 404 returned error can't find the container with id 5bcc1e68e02cc00fb92d221602740758f42379c878600a7480da37721a6742c9
Sep 18 16:56:49.600612 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:49.600512    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" event={"ID":"e29dd90d-c910-42a0-bb83-293d428d4755","Type":"ContainerStarted","Data":"531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd"}
Sep 18 16:56:49.600612 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:49.600574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" event={"ID":"e29dd90d-c910-42a0-bb83-293d428d4755","Type":"ContainerStarted","Data":"5bcc1e68e02cc00fb92d221602740758f42379c878600a7480da37721a6742c9"}
Sep 18 16:56:49.620767 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:49.620708    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" podStartSLOduration=1.620688793 podStartE2EDuration="1.620688793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:56:49.618759086 +0000 UTC m=+2443.394191260" watchObservedRunningTime="2026-09-18 16:56:49.620688793 +0000 UTC m=+2443.396120955"
Sep 18 16:56:49.998434 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:49.998391    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:55.519839 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:55.519800    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:56:55.528027 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:55.527992    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:56:58.997966 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:58.997907    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:59.012138 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:59.012110    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:56:59.668047 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:56:59.668006    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:57:22.411245 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:22.411192    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"]
Sep 18 16:57:22.829095 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:22.828990    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"]
Sep 18 16:57:22.839291 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:57:22.839221    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 16:57:22.839291 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:57:22.839291 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:57:22.839291 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd" cmd=["/bin/sleep","15"]
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:22.839319    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" message="" err=<
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:57:22.839339    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" podUID="e29dd90d-c910-42a0-bb83-293d428d4755" containerName="main" containerID="cri-o://531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd"
Sep 18 16:57:22.839658 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:22.839398    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" podUID="e29dd90d-c910-42a0-bb83-293d428d4755" containerName="main" containerID="cri-o://531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd" gracePeriod=60
Sep 18 16:57:23.192252 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.192207    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:57:23.305063 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dd90d-c910-42a0-bb83-293d428d4755-tls-certs\") pod \"e29dd90d-c910-42a0-bb83-293d428d4755\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") "
Sep 18 16:57:23.305290 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305129    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vf6xc\" (UniqueName: \"kubernetes.io/projected/e29dd90d-c910-42a0-bb83-293d428d4755-kube-api-access-vf6xc\") pod \"e29dd90d-c910-42a0-bb83-293d428d4755\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") "
Sep 18 16:57:23.305290 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305178    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-tmp-dir\") pod \"e29dd90d-c910-42a0-bb83-293d428d4755\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") "
Sep 18 16:57:23.305290 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-model-cache\") pod \"e29dd90d-c910-42a0-bb83-293d428d4755\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") "
Sep 18 16:57:23.305472 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305305    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-home\") pod \"e29dd90d-c910-42a0-bb83-293d428d4755\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") "
Sep 18 16:57:23.305472 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305371    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-dshm\") pod \"e29dd90d-c910-42a0-bb83-293d428d4755\" (UID: \"e29dd90d-c910-42a0-bb83-293d428d4755\") "
Sep 18 16:57:23.305569 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305492    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-model-cache" (OuterVolumeSpecName: "model-cache") pod "e29dd90d-c910-42a0-bb83-293d428d4755" (UID: "e29dd90d-c910-42a0-bb83-293d428d4755"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:23.305569 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305502    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e29dd90d-c910-42a0-bb83-293d428d4755" (UID: "e29dd90d-c910-42a0-bb83-293d428d4755"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:23.305656 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305594    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-home" (OuterVolumeSpecName: "home") pod "e29dd90d-c910-42a0-bb83-293d428d4755" (UID: "e29dd90d-c910-42a0-bb83-293d428d4755"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:23.305781 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305763    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:23.305848 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305787    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:23.305848 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.305799    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:23.307502 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.307475    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-dshm" (OuterVolumeSpecName: "dshm") pod "e29dd90d-c910-42a0-bb83-293d428d4755" (UID: "e29dd90d-c910-42a0-bb83-293d428d4755"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:23.307620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.307537    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29dd90d-c910-42a0-bb83-293d428d4755-kube-api-access-vf6xc" (OuterVolumeSpecName: "kube-api-access-vf6xc") pod "e29dd90d-c910-42a0-bb83-293d428d4755" (UID: "e29dd90d-c910-42a0-bb83-293d428d4755"). InnerVolumeSpecName "kube-api-access-vf6xc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:57:23.307620 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.307551    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29dd90d-c910-42a0-bb83-293d428d4755-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e29dd90d-c910-42a0-bb83-293d428d4755" (UID: "e29dd90d-c910-42a0-bb83-293d428d4755"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:57:23.406928 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.406896    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf6xc\" (UniqueName: \"kubernetes.io/projected/e29dd90d-c910-42a0-bb83-293d428d4755-kube-api-access-vf6xc\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:23.406928 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.406925    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e29dd90d-c910-42a0-bb83-293d428d4755-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:23.407136 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.406938    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e29dd90d-c910-42a0-bb83-293d428d4755-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:23.797682 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.797587    2562 generic.go:356] "Generic (PLEG): container finished" podID="e29dd90d-c910-42a0-bb83-293d428d4755" containerID="531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd" exitCode=0
Sep 18 16:57:23.797682 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.797638    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"
Sep 18 16:57:23.797682 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.797672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" event={"ID":"e29dd90d-c910-42a0-bb83-293d428d4755","Type":"ContainerDied","Data":"531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd"}
Sep 18 16:57:23.798194 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.797731    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4" event={"ID":"e29dd90d-c910-42a0-bb83-293d428d4755","Type":"ContainerDied","Data":"5bcc1e68e02cc00fb92d221602740758f42379c878600a7480da37721a6742c9"}
Sep 18 16:57:23.798194 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.797762    2562 scope.go:117] "RemoveContainer" containerID="531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd"
Sep 18 16:57:23.808437 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.808416    2562 scope.go:117] "RemoveContainer" containerID="531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd"
Sep 18 16:57:23.808800 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:57:23.808780    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd\": container with ID starting with 531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd not found: ID does not exist" containerID="531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd"
Sep 18 16:57:23.808848 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.808810    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd"} err="failed to get container status \"531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd\": rpc error: code = NotFound desc = could not find container \"531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd\": container with ID starting with 531dc8802b167c557dd5bdd4fc24774a8d14b6f02071fda6d38f1f6d9b6973cd not found: ID does not exist"
Sep 18 16:57:23.820690 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.820656    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"]
Sep 18 16:57:23.826758 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:23.826730    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995j2nk4"]
Sep 18 16:57:24.335097 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:24.335061    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29dd90d-c910-42a0-bb83-293d428d4755" path="/var/lib/kubelet/pods/e29dd90d-c910-42a0-bb83-293d428d4755/volumes"
Sep 18 16:57:37.434906 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:37.434860    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" containerID="cri-o://1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3" gracePeriod=45
Sep 18 16:57:37.520665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:37.520616    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Sep 18 16:57:38.196183 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.196153    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:57:38.241835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.241741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kserve-provision-location\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.241835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.241795    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-cabundle-cert\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.241835 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.241818    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tls-certs\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.242128 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242011    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9rwt\" (UniqueName: \"kubernetes.io/projected/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kube-api-access-m9rwt\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.242128 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242066    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tmp-dir\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.242128 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242096    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 16:57:38.242326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242139    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-model-cache\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.242326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242165    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-dshm\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.242326 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242192    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-home\") pod \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\" (UID: \"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c\") "
Sep 18 16:57:38.242555 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242531    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.242831 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-model-cache" (OuterVolumeSpecName: "model-cache") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:38.242959 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.242886    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-home" (OuterVolumeSpecName: "home") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:38.244847 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.244818    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 16:57:38.244847 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.244825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kube-api-access-m9rwt" (OuterVolumeSpecName: "kube-api-access-m9rwt") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "kube-api-access-m9rwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 16:57:38.244992 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.244925    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-dshm" (OuterVolumeSpecName: "dshm") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:38.261681 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.261607    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:38.268159 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.268123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" (UID: "84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 16:57:38.343577 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.343536    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.343577 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.343576    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9rwt\" (UniqueName: \"kubernetes.io/projected/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kube-api-access-m9rwt\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.343727 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.343608    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.343727 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.343619    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.343727 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.343627    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.343727 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.343635    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.343727 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.343642    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 16:57:38.886028 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.885987    2562 generic.go:356] "Generic (PLEG): container finished" podID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerID="1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3" exitCode=0
Sep 18 16:57:38.886450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.886046    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"
Sep 18 16:57:38.886450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.886067    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" event={"ID":"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c","Type":"ContainerDied","Data":"1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3"}
Sep 18 16:57:38.886450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.886127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4" event={"ID":"84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c","Type":"ContainerDied","Data":"56107cb1bddbd1406e3be5ad55f2401de59bb06998c2cea5bfb6b579b3db8947"}
Sep 18 16:57:38.886450 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.886150    2562 scope.go:117] "RemoveContainer" containerID="1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3"
Sep 18 16:57:38.895280 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.895260    2562 scope.go:117] "RemoveContainer" containerID="c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e"
Sep 18 16:57:38.903935 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.903910    2562 scope.go:117] "RemoveContainer" containerID="1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3"
Sep 18 16:57:38.904468 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:57:38.904439    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3\": container with ID starting with 1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3 not found: ID does not exist" containerID="1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3"
Sep 18 16:57:38.904689 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.904479    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3"} err="failed to get container status \"1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3\": rpc error: code = NotFound desc = could not find container \"1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3\": container with ID starting with 1a492fb8e3ac832bdd33731d822666ff5a2887e190a70bccea27e60d0186d2b3 not found: ID does not exist"
Sep 18 16:57:38.904689 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.904505    2562 scope.go:117] "RemoveContainer" containerID="c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e"
Sep 18 16:57:38.904848 ip-10-0-130-101 kubenswrapper[2562]: E0918 16:57:38.904824    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e\": container with ID starting with c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e not found: ID does not exist" containerID="c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e"
Sep 18 16:57:38.904901 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.904850    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e"} err="failed to get container status \"c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e\": rpc error: code = NotFound desc = could not find container \"c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e\": container with ID starting with c71edd4bbceeaa2118282dcc6510a2f543fb3174e92454c3cd2116e0c14c0e5e not found: ID does not exist"
Sep 18 16:57:38.906737 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.906711    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"]
Sep 18 16:57:38.909976 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:38.909951    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-54f6b74bc-z77b4"]
Sep 18 16:57:40.335052 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:57:40.335018    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" path="/var/lib/kubelet/pods/84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c/volumes"
Sep 18 16:58:19.885929 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.885893    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"]
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886275    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="storage-initializer"
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886289    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="storage-initializer"
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886298    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main"
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886304    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main"
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886318    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29dd90d-c910-42a0-bb83-293d428d4755" containerName="main"
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886323    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29dd90d-c910-42a0-bb83-293d428d4755" containerName="main"
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886388    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="84d9ca22-43c5-4144-a0bb-0a5aa0e43a5c" containerName="main"
Sep 18 16:58:19.886412 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.886398    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29dd90d-c910-42a0-bb83-293d428d4755" containerName="main"
Sep 18 16:58:19.890721 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.890700    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:19.893469 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.893442    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 18 16:58:19.894541 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.894519    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-7plr4\""
Sep 18 16:58:19.894810 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.894786    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 18 16:58:19.895451 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.895407    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 18 16:58:19.900407 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.900375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:19.900534 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.900440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:19.900534 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.900483    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:19.900534 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.900520    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:19.900722 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.900545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbqms\" (UniqueName: \"kubernetes.io/projected/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-kube-api-access-dbqms\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:19.900722 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.900574    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:19.903122 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:19.903098    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"]
Sep 18 16:58:20.001199 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001199 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001206    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001515 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001230    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001515 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001285    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001515 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001308    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbqms\" (UniqueName: \"kubernetes.io/projected/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-kube-api-access-dbqms\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001515 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001738 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001675    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001738 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.001827 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.001782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.003642 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.003619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.003955 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.003935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.011886 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.011857    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbqms\" (UniqueName: \"kubernetes.io/projected/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-kube-api-access-dbqms\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.204294 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.204169    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:20.484429 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:20.484398    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"]
Sep 18 16:58:20.486565 ip-10-0-130-101 kubenswrapper[2562]: W0918 16:58:20.486533    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f9e901_6bb0_4608_8b5c_b1a1f9cc913f.slice/crio-304e9cc33062d7a4001435bbe3189ce4bba61974f7f2eff47a9063f9be8ca7b4 WatchSource:0}: Error finding container 304e9cc33062d7a4001435bbe3189ce4bba61974f7f2eff47a9063f9be8ca7b4: Status 404 returned error can't find the container with id 304e9cc33062d7a4001435bbe3189ce4bba61974f7f2eff47a9063f9be8ca7b4
Sep 18 16:58:21.114742 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:21.114704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" event={"ID":"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f","Type":"ContainerStarted","Data":"370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d"}
Sep 18 16:58:21.115282 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:21.114785    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" event={"ID":"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f","Type":"ContainerStarted","Data":"304e9cc33062d7a4001435bbe3189ce4bba61974f7f2eff47a9063f9be8ca7b4"}
Sep 18 16:58:21.134958 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:21.134895    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" podStartSLOduration=2.134877402 podStartE2EDuration="2.134877402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 16:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 16:58:21.132053038 +0000 UTC m=+2534.907485200" watchObservedRunningTime="2026-09-18 16:58:21.134877402 +0000 UTC m=+2534.910309564"
Sep 18 16:58:21.204665 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:21.204622    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:30.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:30.204913    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:30.217637 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:30.217607    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 16:58:31.179589 ip-10-0-130-101 kubenswrapper[2562]: I0918 16:58:31.179554    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 17:00:43.162768 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.162730    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"]
Sep 18 17:00:43.174953 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:00:43.174894    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 17:00:43.174953 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:00:43.174953 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 17:00:43.174953 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d" cmd=["/bin/sleep","15"]
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.174976    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" message="" err=<
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:00:43.174994    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" podUID="e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" containerName="main" containerID="cri-o://370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d"
Sep 18 17:00:43.175328 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.175055    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" podUID="e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" containerName="main" containerID="cri-o://370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d" gracePeriod=60
Sep 18 17:00:43.527042 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.527016    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 17:00:43.624325 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624284    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tls-certs\") pod \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") "
Sep 18 17:00:43.624325 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624330    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbqms\" (UniqueName: \"kubernetes.io/projected/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-kube-api-access-dbqms\") pod \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") "
Sep 18 17:00:43.624583 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624361    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-model-cache\") pod \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") "
Sep 18 17:00:43.624583 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624378    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-home\") pod \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") "
Sep 18 17:00:43.624583 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624413    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-dshm\") pod \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") "
Sep 18 17:00:43.624583 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624438    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tmp-dir\") pod \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\" (UID: \"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f\") "
Sep 18 17:00:43.624803 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624654    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" (UID: "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:43.624803 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624686    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-home" (OuterVolumeSpecName: "home") pod "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" (UID: "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:43.624803 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.624744    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" (UID: "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:43.626711 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.626665    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-dshm" (OuterVolumeSpecName: "dshm") pod "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" (UID: "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:00:43.626839 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.626738    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" (UID: "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:00:43.626839 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.626755    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-kube-api-access-dbqms" (OuterVolumeSpecName: "kube-api-access-dbqms") pod "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" (UID: "e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f"). InnerVolumeSpecName "kube-api-access-dbqms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:00:43.725939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.725839    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.725939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.725874    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.725939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.725885    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.725939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.725894    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbqms\" (UniqueName: \"kubernetes.io/projected/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-kube-api-access-dbqms\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.725939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.725905    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.725939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.725913    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:00:43.889722 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.889688    2562 generic.go:356] "Generic (PLEG): container finished" podID="e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" containerID="370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d" exitCode=0
Sep 18 17:00:43.889921 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.889734    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"
Sep 18 17:00:43.889921 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.889771    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" event={"ID":"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f","Type":"ContainerDied","Data":"370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d"}
Sep 18 17:00:43.889921 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.889828    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf" event={"ID":"e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f","Type":"ContainerDied","Data":"304e9cc33062d7a4001435bbe3189ce4bba61974f7f2eff47a9063f9be8ca7b4"}
Sep 18 17:00:43.889921 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.889852    2562 scope.go:117] "RemoveContainer" containerID="370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d"
Sep 18 17:00:43.900329 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.900305    2562 scope.go:117] "RemoveContainer" containerID="370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d"
Sep 18 17:00:43.900656 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:00:43.900633    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d\": container with ID starting with 370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d not found: ID does not exist" containerID="370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d"
Sep 18 17:00:43.900729 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.900664    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d"} err="failed to get container status \"370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d\": rpc error: code = NotFound desc = could not find container \"370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d\": container with ID starting with 370c78b84deee7aeeb332ea0cef95b0436ba83699d7a3ae68dd348028945789d not found: ID does not exist"
Sep 18 17:00:43.920592 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.916695    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"]
Sep 18 17:00:43.923303 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:43.922664    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997f4hbf"]
Sep 18 17:00:44.334495 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:00:44.334462    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" path="/var/lib/kubelet/pods/e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f/volumes"
Sep 18 17:01:04.570408 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.570362    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"]
Sep 18 17:01:04.570982 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.570902    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" containerName="main"
Sep 18 17:01:04.570982 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.570920    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" containerName="main"
Sep 18 17:01:04.571104 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.571024    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0f9e901-6bb0-4608-8b5c-b1a1f9cc913f" containerName="main"
Sep 18 17:01:04.576170 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.576145    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.579098 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.579002    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 18 17:01:04.579098 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.579061    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 18 17:01:04.579098 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.579072    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:01:04.579398 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.579121    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 18 17:01:04.579398 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.579291    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-88fl5\""
Sep 18 17:01:04.579506 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.579405    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 18 17:01:04.592442 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.592389    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"]
Sep 18 17:01:04.694993 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.694958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.694993 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.694996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.695197 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.695020    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.695197 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.695129    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.695284 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.695217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.695284 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.695264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7bb\" (UniqueName: \"kubernetes.io/projected/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kube-api-access-5h7bb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.695354 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.695292    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.695354 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.695318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796159 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796116    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796159 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796417 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796199    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796417 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796488 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796460    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796526 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796512    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7bb\" (UniqueName: \"kubernetes.io/projected/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kube-api-access-5h7bb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796576 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796630 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796604    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796709 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796781 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796758    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.796939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796913    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.797011 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.796973    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.797060 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.797043    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.798632 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.798606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.798966 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.798948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.805151 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.805122    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7bb\" (UniqueName: \"kubernetes.io/projected/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kube-api-access-5h7bb\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:04.891315 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:04.891281    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:05.180385 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:05.180354    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"]
Sep 18 17:01:05.182210 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:01:05.182177    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986150ec_3f47_4bb6_9fd2_4e3bb4f98af4.slice/crio-4712e5d9356c6f98a57c9a675bf70df9fc507bf9c4260b7d45016b884f267b4e WatchSource:0}: Error finding container 4712e5d9356c6f98a57c9a675bf70df9fc507bf9c4260b7d45016b884f267b4e: Status 404 returned error can't find the container with id 4712e5d9356c6f98a57c9a675bf70df9fc507bf9c4260b7d45016b884f267b4e
Sep 18 17:01:06.007542 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:06.007504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" event={"ID":"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4","Type":"ContainerStarted","Data":"de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812"}
Sep 18 17:01:06.007997 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:06.007564    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" event={"ID":"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4","Type":"ContainerStarted","Data":"4712e5d9356c6f98a57c9a675bf70df9fc507bf9c4260b7d45016b884f267b4e"}
Sep 18 17:01:08.020795 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:08.020699    2562 generic.go:356] "Generic (PLEG): container finished" podID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerID="de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812" exitCode=0
Sep 18 17:01:08.020795 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:08.020775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" event={"ID":"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4","Type":"ContainerDied","Data":"de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812"}
Sep 18 17:01:09.031212 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:09.031173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" event={"ID":"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4","Type":"ContainerStarted","Data":"214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7"}
Sep 18 17:01:09.056295 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:09.056219    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podStartSLOduration=2.658581156 podStartE2EDuration="5.056202651s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.397621495s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:01:09.053950832 +0000 UTC m=+2702.829383038" watchObservedRunningTime="2026-09-18 17:01:09.056202651 +0000 UTC m=+2702.831634813"
Sep 18 17:01:09.891814 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:09.891771    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:14.892459 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:14.892413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:01:14.894207 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:14.894169    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:01:24.892090 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:24.892029    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:01:34.892515 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:34.892453    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:01:44.892198 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:44.892151    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:01:54.892082 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:01:54.892015    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:02:04.892657 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:02:04.892602    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:02:14.892617 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:02:14.892570    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:02:24.892309 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:02:24.892264    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:02:34.891847 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:02:34.891790    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:02:44.891774 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:02:44.891729    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:02:54.892169 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:02:54.892116    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:03:04.892732 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:04.892685    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:03:14.902455 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:14.902413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:03:14.910737 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:14.910699    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:03:19.717229 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:19.717192    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"]
Sep 18 17:03:34.740682 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:34.740614    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" containerID="cri-o://214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7" gracePeriod=45
Sep 18 17:03:34.903667 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:34.903620    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Sep 18 17:03:35.491920 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.491891    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:03:35.525975 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.525945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-dshm\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526171 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526010    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-model-cache\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526171 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h7bb\" (UniqueName: \"kubernetes.io/projected/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kube-api-access-5h7bb\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526171 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526069    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-cabundle-cert\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526171 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526085    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kserve-provision-location\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526171 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526127    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-home\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526171 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tmp-dir\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526509 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526193    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tls-certs\") pod \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\" (UID: \"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4\") "
Sep 18 17:03:35.526509 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526322    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-model-cache" (OuterVolumeSpecName: "model-cache") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:03:35.526509 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526479    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.526672 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526592    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:03:35.526960 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.526930    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-home" (OuterVolumeSpecName: "home") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:03:35.528617 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.528580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kube-api-access-5h7bb" (OuterVolumeSpecName: "kube-api-access-5h7bb") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "kube-api-access-5h7bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:03:35.529121 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.529101    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:03:35.529197 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.529154    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-dshm" (OuterVolumeSpecName: "dshm") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:03:35.545301 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.545250    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:03:35.554099 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.554058    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" (UID: "986150ec-3f47-4bb6-9fd2-4e3bb4f98af4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:03:35.627376 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.627335    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.627376 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.627366    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.627376 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.627377    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7bb\" (UniqueName: \"kubernetes.io/projected/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kube-api-access-5h7bb\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.627376 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.627386    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.627651 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.627395    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.627651 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.627404    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.627651 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.627412    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:03:35.889305 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.889266    2562 generic.go:356] "Generic (PLEG): container finished" podID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerID="214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7" exitCode=0
Sep 18 17:03:35.889742 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.889323    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"
Sep 18 17:03:35.889742 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.889351    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" event={"ID":"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4","Type":"ContainerDied","Data":"214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7"}
Sep 18 17:03:35.889742 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.889404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht" event={"ID":"986150ec-3f47-4bb6-9fd2-4e3bb4f98af4","Type":"ContainerDied","Data":"4712e5d9356c6f98a57c9a675bf70df9fc507bf9c4260b7d45016b884f267b4e"}
Sep 18 17:03:35.889742 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.889429    2562 scope.go:117] "RemoveContainer" containerID="214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7"
Sep 18 17:03:35.899491 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.899463    2562 scope.go:117] "RemoveContainer" containerID="de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812"
Sep 18 17:03:35.908140 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.908121    2562 scope.go:117] "RemoveContainer" containerID="214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7"
Sep 18 17:03:35.908483 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:03:35.908463    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7\": container with ID starting with 214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7 not found: ID does not exist" containerID="214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7"
Sep 18 17:03:35.908536 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.908493    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7"} err="failed to get container status \"214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7\": rpc error: code = NotFound desc = could not find container \"214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7\": container with ID starting with 214b1e4646a0944d4e3d6e8818eef4d50258250624f143e57d4372a3aef7c4c7 not found: ID does not exist"
Sep 18 17:03:35.908536 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.908513    2562 scope.go:117] "RemoveContainer" containerID="de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812"
Sep 18 17:03:35.908751 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:03:35.908734    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812\": container with ID starting with de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812 not found: ID does not exist" containerID="de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812"
Sep 18 17:03:35.908797 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.908757    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812"} err="failed to get container status \"de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812\": rpc error: code = NotFound desc = could not find container \"de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812\": container with ID starting with de165ec8818766fe7648a56a836c44e65d2fa4ff9ce58da2a8f462ce73e72812 not found: ID does not exist"
Sep 18 17:03:35.913929 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.913898    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"]
Sep 18 17:03:35.917475 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:35.917442    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6d5d658c94mq7ht"]
Sep 18 17:03:36.334535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:03:36.334449    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" path="/var/lib/kubelet/pods/986150ec-3f47-4bb6-9fd2-4e3bb4f98af4/volumes"
Sep 18 17:04:31.593937 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.593896    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"]
Sep 18 17:04:31.594629 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.594489    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="storage-initializer"
Sep 18 17:04:31.594629 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.594513    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="storage-initializer"
Sep 18 17:04:31.594629 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.594550    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main"
Sep 18 17:04:31.594629 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.594559    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main"
Sep 18 17:04:31.594854 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.594642    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="986150ec-3f47-4bb6-9fd2-4e3bb4f98af4" containerName="main"
Sep 18 17:04:31.599129 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.599104    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.601865 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.601842    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 18 17:04:31.601992 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.601862    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-dwrb8\""
Sep 18 17:04:31.602060 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.601988    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 18 17:04:31.602123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.602079    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 18 17:04:31.607127 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.607100    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.607277 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.607135    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.607277 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.607162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.607277 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.607203    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxcwh\" (UniqueName: \"kubernetes.io/projected/30e41a95-5518-43e1-a735-ee0cfcabde55-kube-api-access-hxcwh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.607277 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.607264    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e41a95-5518-43e1-a735-ee0cfcabde55-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.607486 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.607307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.607486 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.607368    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.611787 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.611759    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"]
Sep 18 17:04:31.707827 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.707786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxcwh\" (UniqueName: \"kubernetes.io/projected/30e41a95-5518-43e1-a735-ee0cfcabde55-kube-api-access-hxcwh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.707827 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.707819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e41a95-5518-43e1-a735-ee0cfcabde55-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708085 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.707852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708085 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.707882    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708085 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.707914    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708085 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.707936    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708085 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.707965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708379 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.708360    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.708393    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708483 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.708453    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.708520 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.708484    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.710279 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.710253    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.710625 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.710606    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e41a95-5518-43e1-a735-ee0cfcabde55-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.716589 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.716568    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxcwh\" (UniqueName: \"kubernetes.io/projected/30e41a95-5518-43e1-a735-ee0cfcabde55-kube-api-access-hxcwh\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:31.909786 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:31.909746    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:32.195989 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:32.195966    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"]
Sep 18 17:04:32.256719 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:32.256684    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 17:04:33.205990 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:33.205939    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerStarted","Data":"8b1f002d0f08866b95ab043e724a0ac3877cf4bd0528d6b08c6efe1737913650"}
Sep 18 17:04:40.249628 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:40.249589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerStarted","Data":"cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1"}
Sep 18 17:04:41.257491 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:41.257397    2562 generic.go:356] "Generic (PLEG): container finished" podID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerID="cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1" exitCode=0
Sep 18 17:04:41.257907 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:41.257484    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerDied","Data":"cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1"}
Sep 18 17:04:42.273083 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:42.273045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerStarted","Data":"d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804"}
Sep 18 17:04:42.273587 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:42.273155    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerStarted","Data":"07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb"}
Sep 18 17:04:42.295622 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:42.295566    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podStartSLOduration=2.720403011 podStartE2EDuration="11.295549887s" totalImagesPullingTime="7.38973787s" totalInitContainerRuntime="1.185409006s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:04:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:04:42.294480242 +0000 UTC m=+2916.069912420" watchObservedRunningTime="2026-09-18 17:04:42.295549887 +0000 UTC m=+2916.070982048"
Sep 18 17:04:42.910123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:42.910081    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:51.910881 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:51.910832    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:04:51.912391 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:04:51.912352    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:05:01.910890 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:05:01.910835    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:05:11.910986 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:05:11.910932    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:05:21.911009 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:05:21.910909    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:05:31.910573 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:05:31.910521    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:05:41.910532 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:05:41.910480    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:05:51.911154 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:05:51.911099    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:06:01.910258 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:01.910192    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:06:11.911191 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:11.911139    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:06:21.910554 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:21.910494    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:06:31.910307 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:31.910264    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:06:41.920753 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:41.920700    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:06:41.929017 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:41.928985    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:06:47.475123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:47.475088    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"]
Sep 18 17:06:47.475583 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:47.475451    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="modelcar" containerID="cri-o://d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804" gracePeriod=60
Sep 18 17:06:48.012100 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:48.012070    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47_30e41a95-5518-43e1-a735-ee0cfcabde55/modelcar/0.log"
Sep 18 17:06:48.012726 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:48.012703    2562 generic.go:356] "Generic (PLEG): container finished" podID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerID="d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804" exitCode=143
Sep 18 17:06:48.012798 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:06:48.012747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerDied","Data":"d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804"}
Sep 18 17:07:02.498098 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:02.498038    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" containerID="cri-o://07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb" gracePeriod=45
Sep 18 17:07:02.921729 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:02.921677    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Sep 18 17:07:03.356637 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.356611    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47_30e41a95-5518-43e1-a735-ee0cfcabde55/modelcar/0.log"
Sep 18 17:07:03.359583 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.359561    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:07:03.540284 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540141    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxcwh\" (UniqueName: \"kubernetes.io/projected/30e41a95-5518-43e1-a735-ee0cfcabde55-kube-api-access-hxcwh\") pod \"30e41a95-5518-43e1-a735-ee0cfcabde55\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") "
Sep 18 17:07:03.540284 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540211    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-model-cache\") pod \"30e41a95-5518-43e1-a735-ee0cfcabde55\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") "
Sep 18 17:07:03.540843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540304    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-kserve-provision-location\") pod \"30e41a95-5518-43e1-a735-ee0cfcabde55\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") "
Sep 18 17:07:03.540843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e41a95-5518-43e1-a735-ee0cfcabde55-tls-certs\") pod \"30e41a95-5518-43e1-a735-ee0cfcabde55\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") "
Sep 18 17:07:03.540843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540393    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-tmp-dir\") pod \"30e41a95-5518-43e1-a735-ee0cfcabde55\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") "
Sep 18 17:07:03.540843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540423    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-dshm\") pod \"30e41a95-5518-43e1-a735-ee0cfcabde55\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") "
Sep 18 17:07:03.540843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540466    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-home\") pod \"30e41a95-5518-43e1-a735-ee0cfcabde55\" (UID: \"30e41a95-5518-43e1-a735-ee0cfcabde55\") "
Sep 18 17:07:03.540843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540519    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-model-cache" (OuterVolumeSpecName: "model-cache") pod "30e41a95-5518-43e1-a735-ee0cfcabde55" (UID: "30e41a95-5518-43e1-a735-ee0cfcabde55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:03.540843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.540580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "30e41a95-5518-43e1-a735-ee0cfcabde55" (UID: "30e41a95-5518-43e1-a735-ee0cfcabde55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:03.541208 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.541181    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-home" (OuterVolumeSpecName: "home") pod "30e41a95-5518-43e1-a735-ee0cfcabde55" (UID: "30e41a95-5518-43e1-a735-ee0cfcabde55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:03.541379 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.541360    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:07:03.541440 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.541382    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:07:03.541440 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.541401    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:07:03.542661 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.542625    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e41a95-5518-43e1-a735-ee0cfcabde55-kube-api-access-hxcwh" (OuterVolumeSpecName: "kube-api-access-hxcwh") pod "30e41a95-5518-43e1-a735-ee0cfcabde55" (UID: "30e41a95-5518-43e1-a735-ee0cfcabde55"). InnerVolumeSpecName "kube-api-access-hxcwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:07:03.542661 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.542646    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e41a95-5518-43e1-a735-ee0cfcabde55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30e41a95-5518-43e1-a735-ee0cfcabde55" (UID: "30e41a95-5518-43e1-a735-ee0cfcabde55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:07:03.543000 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.542971    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-dshm" (OuterVolumeSpecName: "dshm") pod "30e41a95-5518-43e1-a735-ee0cfcabde55" (UID: "30e41a95-5518-43e1-a735-ee0cfcabde55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:03.552387 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.552347    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30e41a95-5518-43e1-a735-ee0cfcabde55" (UID: "30e41a95-5518-43e1-a735-ee0cfcabde55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:07:03.642689 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.642647    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:07:03.642689 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.642680    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e41a95-5518-43e1-a735-ee0cfcabde55-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:07:03.642689 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.642691    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxcwh\" (UniqueName: \"kubernetes.io/projected/30e41a95-5518-43e1-a735-ee0cfcabde55-kube-api-access-hxcwh\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:07:03.642689 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:03.642699    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e41a95-5518-43e1-a735-ee0cfcabde55-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:07:04.099606 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.099573    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47_30e41a95-5518-43e1-a735-ee0cfcabde55/modelcar/0.log"
Sep 18 17:07:04.100358 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.100332    2562 generic.go:356] "Generic (PLEG): container finished" podID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerID="07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb" exitCode=0
Sep 18 17:07:04.100465 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.100402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerDied","Data":"07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb"}
Sep 18 17:07:04.100508 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.100482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47" event={"ID":"30e41a95-5518-43e1-a735-ee0cfcabde55","Type":"ContainerDied","Data":"8b1f002d0f08866b95ab043e724a0ac3877cf4bd0528d6b08c6efe1737913650"}
Sep 18 17:07:04.100508 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.100496    2562 scope.go:117] "RemoveContainer" containerID="d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804"
Sep 18 17:07:04.100573 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.100405    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"
Sep 18 17:07:04.110766 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.110735    2562 scope.go:117] "RemoveContainer" containerID="07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb"
Sep 18 17:07:04.120256 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.120214    2562 scope.go:117] "RemoveContainer" containerID="cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1"
Sep 18 17:07:04.125690 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.125657    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"]
Sep 18 17:07:04.129123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.129077    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5d78c47"]
Sep 18 17:07:04.133996 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.133970    2562 scope.go:117] "RemoveContainer" containerID="d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804"
Sep 18 17:07:04.134376 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:07:04.134354    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804\": container with ID starting with d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804 not found: ID does not exist" containerID="d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804"
Sep 18 17:07:04.134446 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.134386    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804"} err="failed to get container status \"d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804\": rpc error: code = NotFound desc = could not find container \"d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804\": container with ID starting with d1add00065609503a15ade1d8b2c651df36c64d18c724e55ed919dad3c8e8804 not found: ID does not exist"
Sep 18 17:07:04.134446 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.134405    2562 scope.go:117] "RemoveContainer" containerID="07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb"
Sep 18 17:07:04.134692 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:07:04.134675    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb\": container with ID starting with 07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb not found: ID does not exist" containerID="07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb"
Sep 18 17:07:04.134735 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.134699    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb"} err="failed to get container status \"07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb\": rpc error: code = NotFound desc = could not find container \"07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb\": container with ID starting with 07ffe7837dc1ad7132afd5641006252be3ad79db11b7779c6c78e1fae245a1cb not found: ID does not exist"
Sep 18 17:07:04.134735 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.134715    2562 scope.go:117] "RemoveContainer" containerID="cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1"
Sep 18 17:07:04.135048 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:07:04.135026    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1\": container with ID starting with cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1 not found: ID does not exist" containerID="cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1"
Sep 18 17:07:04.135112 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.135057    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1"} err="failed to get container status \"cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1\": rpc error: code = NotFound desc = could not find container \"cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1\": container with ID starting with cfc0a4986c212ee130dcb96a798e581def4d8a1b18669c938fb8e966f279bbb1 not found: ID does not exist"
Sep 18 17:07:04.334327 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:07:04.334283    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" path="/var/lib/kubelet/pods/30e41a95-5518-43e1-a735-ee0cfcabde55/volumes"
Sep 18 17:08:25.204447 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204408    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"]
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204763    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="modelcar-init"
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204775    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="modelcar-init"
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204786    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main"
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204791    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main"
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204799    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="modelcar"
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204804    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="modelcar"
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204865    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="main"
Sep 18 17:08:25.204955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.204873    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e41a95-5518-43e1-a735-ee0cfcabde55" containerName="modelcar"
Sep 18 17:08:25.208137 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.208120    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.210864 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.210837    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 18 17:08:25.211002 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.210879    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:08:25.211097 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.211081    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 18 17:08:25.211180 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.211158    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 18 17:08:25.211262 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.211169    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-qc6zq\""
Sep 18 17:08:25.211318 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.211256    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 18 17:08:25.219920 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.219894    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"]
Sep 18 17:08:25.291742 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291700    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfdnf\" (UniqueName: \"kubernetes.io/projected/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kube-api-access-vfdnf\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.291972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291751    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.291972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291772    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.291972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291818    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-home\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.291972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.291972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291904    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.291972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291971    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-dshm\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.292279 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.291991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.392843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.392800    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfdnf\" (UniqueName: \"kubernetes.io/projected/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kube-api-access-vfdnf\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.392843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.392850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393135 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.392869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393135 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.392887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-home\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393135 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.392912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393135 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.392930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393135 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.392989    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-dshm\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393135 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.393017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393464 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.393372    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tmp-dir\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393464 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.393405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-model-cache\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393464 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.393437    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kserve-provision-location\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393617 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.393534    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-home\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.393770 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.393750    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-cabundle-cert\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.395455 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.395433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-dshm\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.395713 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.395696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tls-certs\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.405032 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.404998    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfdnf\" (UniqueName: \"kubernetes.io/projected/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kube-api-access-vfdnf\") pod \"stop-feature-test-kserve-6bfc568595-l6gzv\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.519484 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.519401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:25.800839 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:25.800813    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"]
Sep 18 17:08:25.803362 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:08:25.803332    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430bbf30_0215_4ede_bde3_dd8d4fa63cfb.slice/crio-8aedae7f2f7f7890a22b29f320a08b2e87518413079861f2a8efafd46b50fd57 WatchSource:0}: Error finding container 8aedae7f2f7f7890a22b29f320a08b2e87518413079861f2a8efafd46b50fd57: Status 404 returned error can't find the container with id 8aedae7f2f7f7890a22b29f320a08b2e87518413079861f2a8efafd46b50fd57
Sep 18 17:08:26.549514 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:26.549476    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" event={"ID":"430bbf30-0215-4ede-bde3-dd8d4fa63cfb","Type":"ContainerStarted","Data":"aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f"}
Sep 18 17:08:26.549923 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:26.549537    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" event={"ID":"430bbf30-0215-4ede-bde3-dd8d4fa63cfb","Type":"ContainerStarted","Data":"8aedae7f2f7f7890a22b29f320a08b2e87518413079861f2a8efafd46b50fd57"}
Sep 18 17:08:28.565493 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:28.565404    2562 generic.go:356] "Generic (PLEG): container finished" podID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerID="aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f" exitCode=0
Sep 18 17:08:28.565493 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:28.565473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" event={"ID":"430bbf30-0215-4ede-bde3-dd8d4fa63cfb","Type":"ContainerDied","Data":"aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f"}
Sep 18 17:08:29.573762 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:29.573730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" event={"ID":"430bbf30-0215-4ede-bde3-dd8d4fa63cfb","Type":"ContainerStarted","Data":"1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb"}
Sep 18 17:08:29.601927 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:29.601867    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podStartSLOduration=2.191228973 podStartE2EDuration="4.601849828s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.410620855s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:08:29.598996557 +0000 UTC m=+3143.374428742" watchObservedRunningTime="2026-09-18 17:08:29.601849828 +0000 UTC m=+3143.377282030"
Sep 18 17:08:30.519683 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:30.519643    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:35.519866 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:35.519806    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:08:35.521744 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:35.521712    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:08:45.520834 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:45.520783    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:08:55.520228 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:08:55.520167    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:09:05.519961 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:09:05.519907    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:09:15.520640 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:09:15.520594    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:09:25.520049 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:09:25.519994    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:09:35.519945 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:09:35.519883    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:09:45.520224 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:09:45.520170    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:09:55.520483 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:09:55.520437    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:10:05.520667 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:05.520627    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:10:15.520437 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:15.520391    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:10:25.520686 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:25.520641    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:10:35.530853 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:35.530823    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:10:35.539231 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:35.539200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:10:38.054403 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:38.054359    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"]
Sep 18 17:10:53.077875 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.077826    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" containerID="cri-o://1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb" gracePeriod=45
Sep 18 17:10:53.531688 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.531644    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Sep 18 17:10:53.831517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.831486    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:10:53.890636 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890592    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfdnf\" (UniqueName: \"kubernetes.io/projected/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kube-api-access-vfdnf\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.890838 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890651    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kserve-provision-location\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.890838 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tls-certs\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.890838 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890728    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-home\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.890838 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890755    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-dshm\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.890838 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890798    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-cabundle-cert\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.890838 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tmp-dir\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.891148 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.890896    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-model-cache\") pod \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\" (UID: \"430bbf30-0215-4ede-bde3-dd8d4fa63cfb\") "
Sep 18 17:10:53.891405 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.891377    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-model-cache" (OuterVolumeSpecName: "model-cache") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:53.891549 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.891517    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-home" (OuterVolumeSpecName: "home") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:53.891642 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.891622    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:10:53.893609 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.893580    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-dshm" (OuterVolumeSpecName: "dshm") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:53.893691 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.893642    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kube-api-access-vfdnf" (OuterVolumeSpecName: "kube-api-access-vfdnf") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "kube-api-access-vfdnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:10:53.893691 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.893659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:10:53.908686 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.908643    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:53.914999 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.914968    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "430bbf30-0215-4ede-bde3-dd8d4fa63cfb" (UID: "430bbf30-0215-4ede-bde3-dd8d4fa63cfb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:10:53.992287 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992170    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfdnf\" (UniqueName: \"kubernetes.io/projected/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kube-api-access-vfdnf\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:53.992287 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992200    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:53.992287 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992210    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:53.992287 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992221    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:53.992287 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992259    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:53.992287 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992268    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:53.992602 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992298    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:53.992602 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:53.992310    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bbf30-0215-4ede-bde3-dd8d4fa63cfb-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:10:54.414395 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.414358    2562 generic.go:356] "Generic (PLEG): container finished" podID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerID="1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb" exitCode=0
Sep 18 17:10:54.414819 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.414416    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"
Sep 18 17:10:54.414819 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.414437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" event={"ID":"430bbf30-0215-4ede-bde3-dd8d4fa63cfb","Type":"ContainerDied","Data":"1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb"}
Sep 18 17:10:54.414819 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.414498    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv" event={"ID":"430bbf30-0215-4ede-bde3-dd8d4fa63cfb","Type":"ContainerDied","Data":"8aedae7f2f7f7890a22b29f320a08b2e87518413079861f2a8efafd46b50fd57"}
Sep 18 17:10:54.414819 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.414521    2562 scope.go:117] "RemoveContainer" containerID="1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb"
Sep 18 17:10:54.426853 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.426828    2562 scope.go:117] "RemoveContainer" containerID="aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f"
Sep 18 17:10:54.436162 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.436132    2562 scope.go:117] "RemoveContainer" containerID="1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb"
Sep 18 17:10:54.436629 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:10:54.436598    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb\": container with ID starting with 1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb not found: ID does not exist" containerID="1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb"
Sep 18 17:10:54.436756 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.436646    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb"} err="failed to get container status \"1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb\": rpc error: code = NotFound desc = could not find container \"1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb\": container with ID starting with 1565d4f68c246d86d65bd02a3ffae4e7c265de61bbdb1790d8a619c1c7b6ceeb not found: ID does not exist"
Sep 18 17:10:54.436756 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.436676    2562 scope.go:117] "RemoveContainer" containerID="aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f"
Sep 18 17:10:54.437135 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:10:54.437108    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f\": container with ID starting with aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f not found: ID does not exist" containerID="aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f"
Sep 18 17:10:54.437227 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.437141    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f"} err="failed to get container status \"aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f\": rpc error: code = NotFound desc = could not find container \"aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f\": container with ID starting with aaa234cee0080edb7979ee9f53c32402e105516da3b204ae39bd06dd1e75e84f not found: ID does not exist"
Sep 18 17:10:54.437923 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.437900    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"]
Sep 18 17:10:54.441949 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:54.441920    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-6bfc568595-l6gzv"]
Sep 18 17:10:56.335827 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:10:56.335791    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" path="/var/lib/kubelet/pods/430bbf30-0215-4ede-bde3-dd8d4fa63cfb/volumes"
Sep 18 17:11:10.689453 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.689416    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"]
Sep 18 17:11:10.690115 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.689783    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main"
Sep 18 17:11:10.690115 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.689794    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main"
Sep 18 17:11:10.690115 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.689804    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="storage-initializer"
Sep 18 17:11:10.690115 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.689809    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="storage-initializer"
Sep 18 17:11:10.690115 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.689872    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="430bbf30-0215-4ede-bde3-dd8d4fa63cfb" containerName="main"
Sep 18 17:11:10.692928 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.692900    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.695709 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.695685    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:11:10.695839 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.695687    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 18 17:11:10.695978 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.695962    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 18 17:11:10.696286 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.696262    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 18 17:11:10.696407 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.696316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-6jqtg\""
Sep 18 17:11:10.696407 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.696328    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 18 17:11:10.705734 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.705685    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"]
Sep 18 17:11:10.840304 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kserve-provision-location\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.840517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840312    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tls-certs\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.840517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840343    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-cabundle-cert\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.840517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tmp-dir\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.840517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-home\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.840517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840418    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-model-cache\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.840517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6vv\" (UniqueName: \"kubernetes.io/projected/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kube-api-access-9g6vv\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.840517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.840513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-dshm\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941653 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941560    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6vv\" (UniqueName: \"kubernetes.io/projected/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kube-api-access-9g6vv\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941653 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-dshm\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941885 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kserve-provision-location\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941885 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941698    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tls-certs\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941885 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-cabundle-cert\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941885 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tmp-dir\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941885 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941787    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-home\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.941885 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.941809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-model-cache\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.942198 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.942175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tmp-dir\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.942198 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.942190    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kserve-provision-location\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.942336 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.942310    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-home\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.942457 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.942394    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-model-cache\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.942603 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.942588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-cabundle-cert\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.944082 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.944060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-dshm\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.944302 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.944283    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tls-certs\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:10.949851 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:10.949825    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6vv\" (UniqueName: \"kubernetes.io/projected/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kube-api-access-9g6vv\") pod \"router-with-refs-test-kserve-5679bbc469-vczcz\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:11.004677 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:11.004631    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:11.293984 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:11.293955    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"]
Sep 18 17:11:11.296226 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:11:11.296197    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf23faea_db4a_4f4b_a5f5_cdfc531f357e.slice/crio-72f1a74eabb0a959ee47218ab193a9071c4e705298e1aca9ae684ece9a021861 WatchSource:0}: Error finding container 72f1a74eabb0a959ee47218ab193a9071c4e705298e1aca9ae684ece9a021861: Status 404 returned error can't find the container with id 72f1a74eabb0a959ee47218ab193a9071c4e705298e1aca9ae684ece9a021861
Sep 18 17:11:11.514330 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:11.514291    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" event={"ID":"bf23faea-db4a-4f4b-a5f5-cdfc531f357e","Type":"ContainerStarted","Data":"1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2"}
Sep 18 17:11:11.514521 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:11.514363    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" event={"ID":"bf23faea-db4a-4f4b-a5f5-cdfc531f357e","Type":"ContainerStarted","Data":"72f1a74eabb0a959ee47218ab193a9071c4e705298e1aca9ae684ece9a021861"}
Sep 18 17:11:14.533670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:14.533635    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerID="1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2" exitCode=0
Sep 18 17:11:14.534078 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:14.533695    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" event={"ID":"bf23faea-db4a-4f4b-a5f5-cdfc531f357e","Type":"ContainerDied","Data":"1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2"}
Sep 18 17:11:15.541819 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:15.541781    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" event={"ID":"bf23faea-db4a-4f4b-a5f5-cdfc531f357e","Type":"ContainerStarted","Data":"aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059"}
Sep 18 17:11:15.567572 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:15.567515    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podStartSLOduration=3.240802244 podStartE2EDuration="5.567497133s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.326694889s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:11:15.564338395 +0000 UTC m=+3309.339770583" watchObservedRunningTime="2026-09-18 17:11:15.567497133 +0000 UTC m=+3309.342929294"
Sep 18 17:11:16.005177 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:16.005134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:21.005185 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:21.005090    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:11:21.006982 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:21.006952    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:11:31.005171 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:31.005120    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:11:41.005366 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:41.005317    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:11:51.005411 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:11:51.005360    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:12:01.005408 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:12:01.005357    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:12:11.005626 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:12:11.005580    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:12:21.005958 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:12:21.005905    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:12:31.005450 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:12:31.005404    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:12:41.005908 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:12:41.005853    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:12:51.005145 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:12:51.005051    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:13:01.005847 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:01.005787    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:13:11.005676 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:11.005625    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Sep 18 17:13:21.016053 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:21.016011    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:13:21.025361 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:21.025327    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:13:26.147618 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:26.147578    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"]
Sep 18 17:13:41.171428 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:41.171367    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main" containerID="cri-o://aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059" gracePeriod=45
Sep 18 17:13:41.926988 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:41.926963    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:13:42.062657 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062563    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kserve-provision-location\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.062657 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062609    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-dshm\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.062657 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062638    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-cabundle-cert\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.062955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062666    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-home\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.062955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g6vv\" (UniqueName: \"kubernetes.io/projected/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kube-api-access-9g6vv\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.062955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062714    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tmp-dir\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.062955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062779    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-model-cache\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.062955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.062818    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tls-certs\") pod \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\" (UID: \"bf23faea-db4a-4f4b-a5f5-cdfc531f357e\") "
Sep 18 17:13:42.063203 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.063031    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:13:42.063203 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.063059    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:42.063484 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.063454    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-home" (OuterVolumeSpecName: "home") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:42.065204 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.065180    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:13:42.065204 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.065187    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kube-api-access-9g6vv" (OuterVolumeSpecName: "kube-api-access-9g6vv") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "kube-api-access-9g6vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:13:42.065204 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.065193    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-dshm" (OuterVolumeSpecName: "dshm") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:42.080618 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.080565    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:42.092570 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.092520    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf23faea-db4a-4f4b-a5f5-cdfc531f357e" (UID: "bf23faea-db4a-4f4b-a5f5-cdfc531f357e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:13:42.164093 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164049    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.164093 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164088    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.164093 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164099    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.164353 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164107    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.164353 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164118    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.164353 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164127    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.164353 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164139    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6vv\" (UniqueName: \"kubernetes.io/projected/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-kube-api-access-9g6vv\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.164353 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.164150    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf23faea-db4a-4f4b-a5f5-cdfc531f357e-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:13:42.399250 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.399197    2562 generic.go:356] "Generic (PLEG): container finished" podID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerID="aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059" exitCode=0
Sep 18 17:13:42.399715 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.399266    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"
Sep 18 17:13:42.399715 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.399268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" event={"ID":"bf23faea-db4a-4f4b-a5f5-cdfc531f357e","Type":"ContainerDied","Data":"aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059"}
Sep 18 17:13:42.399715 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.399393    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz" event={"ID":"bf23faea-db4a-4f4b-a5f5-cdfc531f357e","Type":"ContainerDied","Data":"72f1a74eabb0a959ee47218ab193a9071c4e705298e1aca9ae684ece9a021861"}
Sep 18 17:13:42.399715 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.399417    2562 scope.go:117] "RemoveContainer" containerID="aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059"
Sep 18 17:13:42.408892 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.408873    2562 scope.go:117] "RemoveContainer" containerID="1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2"
Sep 18 17:13:42.417951 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.417933    2562 scope.go:117] "RemoveContainer" containerID="aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059"
Sep 18 17:13:42.418138 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.418111    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"]
Sep 18 17:13:42.418333 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:13:42.418313    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059\": container with ID starting with aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059 not found: ID does not exist" containerID="aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059"
Sep 18 17:13:42.418393 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.418344    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059"} err="failed to get container status \"aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059\": rpc error: code = NotFound desc = could not find container \"aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059\": container with ID starting with aa1eefac148717123b0a6da4118b7afec4ccf8e83dbe268092976ebe30fea059 not found: ID does not exist"
Sep 18 17:13:42.418393 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.418363    2562 scope.go:117] "RemoveContainer" containerID="1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2"
Sep 18 17:13:42.418635 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:13:42.418620    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2\": container with ID starting with 1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2 not found: ID does not exist" containerID="1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2"
Sep 18 17:13:42.418699 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.418637    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2"} err="failed to get container status \"1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2\": rpc error: code = NotFound desc = could not find container \"1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2\": container with ID starting with 1cb366b7c51f021515b153f4d2253f546d0d3db9b416738d41076c89d1376db2 not found: ID does not exist"
Sep 18 17:13:42.422191 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:42.422155    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5679bbc469-vczcz"]
Sep 18 17:13:44.334563 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:44.334527    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" path="/var/lib/kubelet/pods/bf23faea-db4a-4f4b-a5f5-cdfc531f357e/volumes"
Sep 18 17:13:56.178044 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.178005    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"]
Sep 18 17:13:56.178774 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.178748    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="storage-initializer"
Sep 18 17:13:56.178965 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.178945    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="storage-initializer"
Sep 18 17:13:56.179092 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.178973    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main"
Sep 18 17:13:56.179092 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.178982    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main"
Sep 18 17:13:56.179092 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.179091    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf23faea-db4a-4f4b-a5f5-cdfc531f357e" containerName="main"
Sep 18 17:13:56.182855 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.182829    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.185517 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.185485    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"seaweedfs-s3-creds\""
Sep 18 17:13:56.185698 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.185587    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Sep 18 17:13:56.185698 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.185492    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"kube-root-ca.crt\""
Sep 18 17:13:56.185698 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.185639    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"openshift-service-ca.crt\""
Sep 18 17:13:56.185907 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.185892    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:13:56.185963 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.185895    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-3cbf5e9c\"/\"default-dockercfg-bxwbm\""
Sep 18 17:13:56.197601 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.197491    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"]
Sep 18 17:13:56.283742 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.283702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf127b-b61d-47df-9792-ea05e2fc98c6-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.283742 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.283741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.284006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.283835    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-model-cache\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.284006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.283871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tls-certs\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.284006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.283901    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-dshm\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.284006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.283945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9p65\" (UniqueName: \"kubernetes.io/projected/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kube-api-access-p9p65\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.284006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.283999    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-home\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.284182 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.284034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tmp-dir\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.384688 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384648    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-home\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.384884 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384699    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tmp-dir\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.384884 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf127b-b61d-47df-9792-ea05e2fc98c6-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.384884 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.384884 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384814    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-model-cache\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.384884 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384839    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tls-certs\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.384884 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-dshm\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.385220 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.384898    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9p65\" (UniqueName: \"kubernetes.io/projected/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kube-api-access-p9p65\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.385220 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.385197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.385334 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.385226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-home\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.385334 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.385323    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tmp-dir\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.385402 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.385366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-model-cache\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.385659 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.385638    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf127b-b61d-47df-9792-ea05e2fc98c6-cabundle-cert\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.387323 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.387299    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-dshm\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.387567 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.387549    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tls-certs\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.395017 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.394979    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9p65\" (UniqueName: \"kubernetes.io/projected/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kube-api-access-p9p65\") pod \"lora-single-adapter-test-kserve-d754d949c-r58r4\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") " pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.494016 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.493919    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:13:56.780356 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:56.780318    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"]
Sep 18 17:13:56.781707 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:13:56.781681    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbf127b_b61d_47df_9792_ea05e2fc98c6.slice/crio-b752e418b19ff524f0e98ca5338b52448e769be4a858a185b5e47702de0e29d0 WatchSource:0}: Error finding container b752e418b19ff524f0e98ca5338b52448e769be4a858a185b5e47702de0e29d0: Status 404 returned error can't find the container with id b752e418b19ff524f0e98ca5338b52448e769be4a858a185b5e47702de0e29d0
Sep 18 17:13:57.489986 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:57.489940    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" event={"ID":"1fbf127b-b61d-47df-9792-ea05e2fc98c6","Type":"ContainerStarted","Data":"f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441"}
Sep 18 17:13:57.490626 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:13:57.490047    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" event={"ID":"1fbf127b-b61d-47df-9792-ea05e2fc98c6","Type":"ContainerStarted","Data":"b752e418b19ff524f0e98ca5338b52448e769be4a858a185b5e47702de0e29d0"}
Sep 18 17:14:00.508251 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:00.508204    2562 generic.go:356] "Generic (PLEG): container finished" podID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerID="f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441" exitCode=0
Sep 18 17:14:00.508644 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:00.508284    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" event={"ID":"1fbf127b-b61d-47df-9792-ea05e2fc98c6","Type":"ContainerDied","Data":"f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441"}
Sep 18 17:14:01.516872 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:01.516830    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" event={"ID":"1fbf127b-b61d-47df-9792-ea05e2fc98c6","Type":"ContainerStarted","Data":"07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771"}
Sep 18 17:14:01.572174 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:01.572114    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podStartSLOduration=2.769019273 podStartE2EDuration="5.572089058s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.803069785s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:01.570651139 +0000 UTC m=+3475.346083332" watchObservedRunningTime="2026-09-18 17:14:01.572089058 +0000 UTC m=+3475.347521220"
Sep 18 17:14:02.494350 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:02.494312    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:14:06.494979 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:06.494925    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:14:06.496615 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:06.496579    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:14:16.494661 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:16.494615    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:14:23.946637 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.946599    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"]
Sep 18 17:14:23.951270 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.951231    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:23.954208 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.954181    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-j4h4t\""
Sep 18 17:14:23.954475 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.954453    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 18 17:14:23.954475 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.954469    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 18 17:14:23.954645 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.954473    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 18 17:14:23.954645 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.954544    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 18 17:14:23.954746 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.954652    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:14:23.967427 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:23.967399    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"]
Sep 18 17:14:24.024327 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024285    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dfb6d86-0508-42d0-9808-aec909f71469-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.024545 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024355    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfb6d86-0508-42d0-9808-aec909f71469-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.024545 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024393    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqw7g\" (UniqueName: \"kubernetes.io/projected/3dfb6d86-0508-42d0-9808-aec909f71469-kube-api-access-wqw7g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.024545 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.024681 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.024681 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024627    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.024681 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024664    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.024795 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.024695    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125316 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125276    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dfb6d86-0508-42d0-9808-aec909f71469-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125531 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfb6d86-0508-42d0-9808-aec909f71469-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125531 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqw7g\" (UniqueName: \"kubernetes.io/projected/3dfb6d86-0508-42d0-9808-aec909f71469-kube-api-access-wqw7g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125531 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125452    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125531 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125531 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125800 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.125800 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.125659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.126058 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.126034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.126146 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.126077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.126146 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.126085    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.126280 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.126144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dfb6d86-0508-42d0-9808-aec909f71469-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.126280 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.126213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.128078 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.128049    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfb6d86-0508-42d0-9808-aec909f71469-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.128078 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.128054    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.135529 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.135503    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqw7g\" (UniqueName: \"kubernetes.io/projected/3dfb6d86-0508-42d0-9808-aec909f71469-kube-api-access-wqw7g\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.263336 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.263222    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:24.547700 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.547672    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"]
Sep 18 17:14:24.550345 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:14:24.550315    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dfb6d86_0508_42d0_9808_aec909f71469.slice/crio-5eceab2222028779ee9dd938e86b187878b7bc9e78dfac43390b05084c58d5c3 WatchSource:0}: Error finding container 5eceab2222028779ee9dd938e86b187878b7bc9e78dfac43390b05084c58d5c3: Status 404 returned error can't find the container with id 5eceab2222028779ee9dd938e86b187878b7bc9e78dfac43390b05084c58d5c3
Sep 18 17:14:24.650280 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.650229    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" event={"ID":"3dfb6d86-0508-42d0-9808-aec909f71469","Type":"ContainerStarted","Data":"2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1"}
Sep 18 17:14:24.650461 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:24.650294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" event={"ID":"3dfb6d86-0508-42d0-9808-aec909f71469","Type":"ContainerStarted","Data":"5eceab2222028779ee9dd938e86b187878b7bc9e78dfac43390b05084c58d5c3"}
Sep 18 17:14:26.494964 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:26.494920    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:14:27.676880 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:27.676846    2562 generic.go:356] "Generic (PLEG): container finished" podID="3dfb6d86-0508-42d0-9808-aec909f71469" containerID="2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1" exitCode=0
Sep 18 17:14:27.677408 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:27.676921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" event={"ID":"3dfb6d86-0508-42d0-9808-aec909f71469","Type":"ContainerDied","Data":"2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1"}
Sep 18 17:14:28.687064 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:28.687023    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" event={"ID":"3dfb6d86-0508-42d0-9808-aec909f71469","Type":"ContainerStarted","Data":"f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21"}
Sep 18 17:14:28.711608 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:28.711543    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podStartSLOduration=3.24184479 podStartE2EDuration="5.711525068s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.469680278s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:14:28.710510667 +0000 UTC m=+3502.485942869" watchObservedRunningTime="2026-09-18 17:14:28.711525068 +0000 UTC m=+3502.486957269"
Sep 18 17:14:34.264441 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:34.264396    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:34.264864 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:34.264485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:14:34.266680 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:34.266627    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:14:36.496295 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:36.496208    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:14:44.264094 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:44.264028    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:14:46.495067 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:46.495019    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:14:54.264036 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:54.263982    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:14:56.494763 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:14:56.494712    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:15:04.264524 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:04.264474    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:15:06.494589 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:06.494528    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:15:14.264045 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:14.263990    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:15:16.494574 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:16.494519    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:15:24.264365 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:24.264310    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:15:26.505943 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:26.505905    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:15:26.514256 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:26.514202    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:15:34.264120 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:34.264063    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:15:44.264579 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:44.264532    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:15:54.264245 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:15:54.264186    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:16:04.264790 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:04.264745    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:16:04.331780 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:04.331744    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-d754d949c-r58r4_1fbf127b-b61d-47df-9792-ea05e2fc98c6/storage-initializer/0.log"
Sep 18 17:16:04.341718 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:04.341687    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-3cbf5e9c_lora-single-adapter-test-kserve-d754d949c-r58r4_1fbf127b-b61d-47df-9792-ea05e2fc98c6/main/0.log"
Sep 18 17:16:05.520177 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:05.520141    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"]
Sep 18 17:16:14.264067 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:14.264014    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:16:20.544928 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:20.544847    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" containerID="cri-o://07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771" gracePeriod=45
Sep 18 17:16:21.507398 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.507341    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Sep 18 17:16:21.595774 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.595738    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:16:21.705849 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.705741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9p65\" (UniqueName: \"kubernetes.io/projected/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kube-api-access-p9p65\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.705849 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.705792    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kserve-provision-location\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.705849 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.705841    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-dshm\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.706146 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.705899    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-home\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.706146 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.705934    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf127b-b61d-47df-9792-ea05e2fc98c6-cabundle-cert\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.706146 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.705982    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-model-cache\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.706146 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.706013    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tmp-dir\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.706146 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.706054    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tls-certs\") pod \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\" (UID: \"1fbf127b-b61d-47df-9792-ea05e2fc98c6\") "
Sep 18 17:16:21.706418 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.706341    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbf127b-b61d-47df-9792-ea05e2fc98c6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:16:21.706538 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.706503    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-home" (OuterVolumeSpecName: "home") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:16:21.706607 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.706523    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:16:21.708563 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.708535    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:16:21.708727 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.708698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-dshm" (OuterVolumeSpecName: "dshm") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:16:21.708727 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.708671    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kube-api-access-p9p65" (OuterVolumeSpecName: "kube-api-access-p9p65") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "kube-api-access-p9p65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:16:21.720390 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.720346    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:16:21.731750 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.731713    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1fbf127b-b61d-47df-9792-ea05e2fc98c6" (UID: "1fbf127b-b61d-47df-9792-ea05e2fc98c6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:16:21.807248 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807208    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:21.807248 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807252    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fbf127b-b61d-47df-9792-ea05e2fc98c6-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:21.807501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807263    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9p65\" (UniqueName: \"kubernetes.io/projected/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kube-api-access-p9p65\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:21.807501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807272    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:21.807501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807280    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:21.807501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807288    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:21.807501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807296    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fbf127b-b61d-47df-9792-ea05e2fc98c6-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:21.807501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:21.807303    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fbf127b-b61d-47df-9792-ea05e2fc98c6-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:16:22.389560 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.389526    2562 generic.go:356] "Generic (PLEG): container finished" podID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerID="07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771" exitCode=0
Sep 18 17:16:22.389752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.389602    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"
Sep 18 17:16:22.389752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.389605    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" event={"ID":"1fbf127b-b61d-47df-9792-ea05e2fc98c6","Type":"ContainerDied","Data":"07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771"}
Sep 18 17:16:22.389752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.389730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4" event={"ID":"1fbf127b-b61d-47df-9792-ea05e2fc98c6","Type":"ContainerDied","Data":"b752e418b19ff524f0e98ca5338b52448e769be4a858a185b5e47702de0e29d0"}
Sep 18 17:16:22.389929 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.389760    2562 scope.go:117] "RemoveContainer" containerID="07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771"
Sep 18 17:16:22.399750 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.399728    2562 scope.go:117] "RemoveContainer" containerID="f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441"
Sep 18 17:16:22.409834 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.409812    2562 scope.go:117] "RemoveContainer" containerID="07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771"
Sep 18 17:16:22.410202 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:16:22.410181    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771\": container with ID starting with 07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771 not found: ID does not exist" containerID="07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771"
Sep 18 17:16:22.410339 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.410214    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771"} err="failed to get container status \"07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771\": rpc error: code = NotFound desc = could not find container \"07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771\": container with ID starting with 07a32498320fab7dea439ccde9c7dbdf47fa18a5126ca5a04c4f7edb0ae35771 not found: ID does not exist"
Sep 18 17:16:22.410339 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.410259    2562 scope.go:117] "RemoveContainer" containerID="f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441"
Sep 18 17:16:22.410533 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:16:22.410515    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441\": container with ID starting with f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441 not found: ID does not exist" containerID="f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441"
Sep 18 17:16:22.410593 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.410537    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441"} err="failed to get container status \"f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441\": rpc error: code = NotFound desc = could not find container \"f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441\": container with ID starting with f9b9c2eba7b1fcfe8c7b87738163c350445a2127d0f38e6f42a5f40b3db9b441 not found: ID does not exist"
Sep 18 17:16:22.410997 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.410974    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"]
Sep 18 17:16:22.415019 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:22.414992    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-3cbf5e9c/lora-single-adapter-test-kserve-d754d949c-r58r4"]
Sep 18 17:16:24.263850 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:24.263785    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:16:24.335145 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:24.335108    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" path="/var/lib/kubelet/pods/1fbf127b-b61d-47df-9792-ea05e2fc98c6/volumes"
Sep 18 17:16:34.264638 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:34.264587    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Sep 18 17:16:44.274267 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:44.274212    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:16:44.282226 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:44.282188    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:16:54.222159 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:16:54.222118    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"]
Sep 18 17:17:09.248865 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:09.248796    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main" containerID="cri-o://f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21" gracePeriod=45
Sep 18 17:17:10.007133 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.007106    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:17:10.135230 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqw7g\" (UniqueName: \"kubernetes.io/projected/3dfb6d86-0508-42d0-9808-aec909f71469-kube-api-access-wqw7g\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135433 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135263    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-dshm\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135433 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135292    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfb6d86-0508-42d0-9808-aec909f71469-tls-certs\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135433 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135313    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-model-cache\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135433 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135351    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-home\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135433 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135382    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dfb6d86-0508-42d0-9808-aec909f71469-cabundle-cert\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135433 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135430    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-kserve-provision-location\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135765 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135500    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-tmp-dir\") pod \"3dfb6d86-0508-42d0-9808-aec909f71469\" (UID: \"3dfb6d86-0508-42d0-9808-aec909f71469\") "
Sep 18 17:17:10.135765 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135639    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-model-cache" (OuterVolumeSpecName: "model-cache") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.135886 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.135775    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.136150 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.136123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dfb6d86-0508-42d0-9808-aec909f71469-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:17:10.136230 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.136150    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-home" (OuterVolumeSpecName: "home") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.138278 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.138250    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dfb6d86-0508-42d0-9808-aec909f71469-kube-api-access-wqw7g" (OuterVolumeSpecName: "kube-api-access-wqw7g") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "kube-api-access-wqw7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:17:10.138396 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.138342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-dshm" (OuterVolumeSpecName: "dshm") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.138396 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.138357    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dfb6d86-0508-42d0-9808-aec909f71469-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:17:10.152276 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.152227    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.160212 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.160172    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3dfb6d86-0508-42d0-9808-aec909f71469" (UID: "3dfb6d86-0508-42d0-9808-aec909f71469"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:17:10.236756 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.236710    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqw7g\" (UniqueName: \"kubernetes.io/projected/3dfb6d86-0508-42d0-9808-aec909f71469-kube-api-access-wqw7g\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.236756 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.236747    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.236756 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.236757    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dfb6d86-0508-42d0-9808-aec909f71469-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.236756 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.236765    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.237034 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.236774    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3dfb6d86-0508-42d0-9808-aec909f71469-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.237034 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.236782    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.237034 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.236790    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dfb6d86-0508-42d0-9808-aec909f71469-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:17:10.667653 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.667617    2562 generic.go:356] "Generic (PLEG): container finished" podID="3dfb6d86-0508-42d0-9808-aec909f71469" containerID="f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21" exitCode=0
Sep 18 17:17:10.668067 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.667673    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"
Sep 18 17:17:10.668067 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.667700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" event={"ID":"3dfb6d86-0508-42d0-9808-aec909f71469","Type":"ContainerDied","Data":"f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21"}
Sep 18 17:17:10.668067 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.667762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc" event={"ID":"3dfb6d86-0508-42d0-9808-aec909f71469","Type":"ContainerDied","Data":"5eceab2222028779ee9dd938e86b187878b7bc9e78dfac43390b05084c58d5c3"}
Sep 18 17:17:10.668067 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.667784    2562 scope.go:117] "RemoveContainer" containerID="f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21"
Sep 18 17:17:10.677419 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.677396    2562 scope.go:117] "RemoveContainer" containerID="2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1"
Sep 18 17:17:10.686851 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.686825    2562 scope.go:117] "RemoveContainer" containerID="f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21"
Sep 18 17:17:10.687317 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:17:10.687289    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21\": container with ID starting with f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21 not found: ID does not exist" containerID="f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21"
Sep 18 17:17:10.687450 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.687326    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21"} err="failed to get container status \"f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21\": rpc error: code = NotFound desc = could not find container \"f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21\": container with ID starting with f3a45603210274366a5668c7d3cc44231c69b644d636d5dadba002b177d4af21 not found: ID does not exist"
Sep 18 17:17:10.687450 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.687353    2562 scope.go:117] "RemoveContainer" containerID="2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1"
Sep 18 17:17:10.687807 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:17:10.687762    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1\": container with ID starting with 2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1 not found: ID does not exist" containerID="2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1"
Sep 18 17:17:10.687924 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.687800    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1"} err="failed to get container status \"2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1\": rpc error: code = NotFound desc = could not find container \"2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1\": container with ID starting with 2cd8216d8b08879bdb46ad97fe1c4ac7e6fe899bc3d65eaa75e8fcc9da04d5c1 not found: ID does not exist"
Sep 18 17:17:10.689562 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.689527    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"]
Sep 18 17:17:10.693028 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:10.693001    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-f7kwhdc"]
Sep 18 17:17:12.335244 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:17:12.335192    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" path="/var/lib/kubelet/pods/3dfb6d86-0508-42d0-9808-aec909f71469/volumes"
Sep 18 17:18:05.367170 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367134    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"]
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367528    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="storage-initializer"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367542    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="storage-initializer"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367553    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="storage-initializer"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367559    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="storage-initializer"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367578    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367584    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367597    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367602    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367664    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dfb6d86-0508-42d0-9808-aec909f71469" containerName="main"
Sep 18 17:18:05.369611 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.367674    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fbf127b-b61d-47df-9792-ea05e2fc98c6" containerName="main"
Sep 18 17:18:05.370939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.370916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.373409 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.373383    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 17:18:05.373536 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.373415    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:18:05.373536 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.373434    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 17:18:05.373807 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.373791    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 17:18:05.373865 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.373819    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 17:18:05.374445 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.374428    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-jlp6s\""
Sep 18 17:18:05.395152 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.395117    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"]
Sep 18 17:18:05.507475 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.507475 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.507752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j569\" (UniqueName: \"kubernetes.io/projected/2a285ac7-0fa6-42af-93de-48a0297423e8-kube-api-access-2j569\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.507752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.507752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.507752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507674    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a285ac7-0fa6-42af-93de-48a0297423e8-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.507752 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a285ac7-0fa6-42af-93de-48a0297423e8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.507955 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.507770    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608211 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a285ac7-0fa6-42af-93de-48a0297423e8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608211 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608511 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608511 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608511 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j569\" (UniqueName: \"kubernetes.io/projected/2a285ac7-0fa6-42af-93de-48a0297423e8-kube-api-access-2j569\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608511 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608729 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608547    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608729 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608579    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a285ac7-0fa6-42af-93de-48a0297423e8-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608729 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608700    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608729 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608723    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608935    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.608972 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.608970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.609346 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.609328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a285ac7-0fa6-42af-93de-48a0297423e8-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.610906 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.610864    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.611124 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.611094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a285ac7-0fa6-42af-93de-48a0297423e8-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.620801 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.620715    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j569\" (UniqueName: \"kubernetes.io/projected/2a285ac7-0fa6-42af-93de-48a0297423e8-kube-api-access-2j569\") pod \"custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.682222 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.682182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:05.968007 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:05.967975    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"]
Sep 18 17:18:05.969963 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:18:05.969928    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a285ac7_0fa6_42af_93de_48a0297423e8.slice/crio-aaf2bd35a61a5fdcbffec97804523fbc75cdefb04d1a1520bcccf609dfa9273b WatchSource:0}: Error finding container aaf2bd35a61a5fdcbffec97804523fbc75cdefb04d1a1520bcccf609dfa9273b: Status 404 returned error can't find the container with id aaf2bd35a61a5fdcbffec97804523fbc75cdefb04d1a1520bcccf609dfa9273b
Sep 18 17:18:06.979282 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:06.979218    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" event={"ID":"2a285ac7-0fa6-42af-93de-48a0297423e8","Type":"ContainerStarted","Data":"0c0dea64052be737dfdd82d8050f5949a734099ef1188e8e4856bddda171bfef"}
Sep 18 17:18:06.979795 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:06.979309    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" event={"ID":"2a285ac7-0fa6-42af-93de-48a0297423e8","Type":"ContainerStarted","Data":"aaf2bd35a61a5fdcbffec97804523fbc75cdefb04d1a1520bcccf609dfa9273b"}
Sep 18 17:18:08.993677 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:08.993643    2562 generic.go:356] "Generic (PLEG): container finished" podID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerID="0c0dea64052be737dfdd82d8050f5949a734099ef1188e8e4856bddda171bfef" exitCode=0
Sep 18 17:18:08.994151 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:08.993723    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" event={"ID":"2a285ac7-0fa6-42af-93de-48a0297423e8","Type":"ContainerDied","Data":"0c0dea64052be737dfdd82d8050f5949a734099ef1188e8e4856bddda171bfef"}
Sep 18 17:18:10.001291 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:10.001231    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" event={"ID":"2a285ac7-0fa6-42af-93de-48a0297423e8","Type":"ContainerStarted","Data":"abc96e1ef7733eeb791eae59814176c04c703f64f1fe80d1054b38b7fec7b3fb"}
Sep 18 17:18:10.025771 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:10.025714    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podStartSLOduration=2.711122836 podStartE2EDuration="5.025692423s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.314569587s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:18:10.02367086 +0000 UTC m=+3723.799103021" watchObservedRunningTime="2026-09-18 17:18:10.025692423 +0000 UTC m=+3723.801124585"
Sep 18 17:18:15.683050 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:15.683006    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:15.683608 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:15.683096    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:18:15.684676 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:15.684647    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:18:25.682993 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:25.682946    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:18:35.683543 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:35.683490    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:18:45.682738 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:45.682689    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:18:55.683840 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:18:55.683780    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:19:05.683563 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:05.683514    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:19:10.847379 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:10.847332    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"]
Sep 18 17:19:10.851133 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:10.851111    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:10.854110 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:10.854069    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 17:19:10.854270 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:10.854121    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 17:19:10.855310 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:10.855290    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 17:19:10.855310 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:10.855306    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-t7p8k\""
Sep 18 17:19:10.859390 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:10.859365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"]
Sep 18 17:19:11.011059 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:11.011015    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmcx\" (UniqueName: \"kubernetes.io/projected/1196a70e-2a3b-46d1-8afa-2bf25e08fcb1-kube-api-access-lxmcx\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:11.112321 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:11.112222    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmcx\" (UniqueName: \"kubernetes.io/projected/1196a70e-2a3b-46d1-8afa-2bf25e08fcb1-kube-api-access-lxmcx\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:11.122808 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:11.122774    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmcx\" (UniqueName: \"kubernetes.io/projected/1196a70e-2a3b-46d1-8afa-2bf25e08fcb1-kube-api-access-lxmcx\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:14.641763 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:14.641721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:14.647557 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:14.647519    2562 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:14.743170 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:14.743129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:14.743360 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:14.743315    2562 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:14.746410 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:14.746370    2562 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") DevicePath \"csi-2ec842590097daf87e68d99522cb2f514630e473ed082d3808c0ec85a9a9a54e\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:14.994184 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:14.994150    2562 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8e8e8e00a2139f55fc05574dd20b4fe91023a1019de5796549727172032e0e2b/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:15.010446 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:15.010405    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"e2e-pvc-lora-download-nnvkq\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:15.062882 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:15.062837    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:15.340588 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:15.340557    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"]
Sep 18 17:19:15.342405 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:19:15.342362    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1196a70e_2a3b_46d1_8afa_2bf25e08fcb1.slice/crio-9fbacac5a1cc7d326a68f33e3655c3909458f194a3d63638a9f31c6ab99b80c2 WatchSource:0}: Error finding container 9fbacac5a1cc7d326a68f33e3655c3909458f194a3d63638a9f31c6ab99b80c2: Status 404 returned error can't find the container with id 9fbacac5a1cc7d326a68f33e3655c3909458f194a3d63638a9f31c6ab99b80c2
Sep 18 17:19:15.379774 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:15.379735    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq" event={"ID":"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1","Type":"ContainerStarted","Data":"9fbacac5a1cc7d326a68f33e3655c3909458f194a3d63638a9f31c6ab99b80c2"}
Sep 18 17:19:15.683018 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:15.682972    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:19:16.390856 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:16.390811    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq" event={"ID":"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1","Type":"ContainerStarted","Data":"2e960fa7bf8ca420f25a1109a6bb7b8a4e1bd5afbf8266b56286a58b72ea42a9"}
Sep 18 17:19:16.407267 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:16.407180    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq" podStartSLOduration=9.407156787 podStartE2EDuration="9.407156787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:19:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:19:16.405371744 +0000 UTC m=+3790.180803932" watchObservedRunningTime="2026-09-18 17:19:16.407156787 +0000 UTC m=+3790.182588950"
Sep 18 17:19:18.407140 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:18.407102    2562 generic.go:356] "Generic (PLEG): container finished" podID="1196a70e-2a3b-46d1-8afa-2bf25e08fcb1" containerID="2e960fa7bf8ca420f25a1109a6bb7b8a4e1bd5afbf8266b56286a58b72ea42a9" exitCode=0
Sep 18 17:19:18.407574 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:18.407181    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq" event={"ID":"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1","Type":"ContainerDied","Data":"2e960fa7bf8ca420f25a1109a6bb7b8a4e1bd5afbf8266b56286a58b72ea42a9"}
Sep 18 17:19:19.582006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:19.581963    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:19.590062 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:19.590026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxmcx\" (UniqueName: \"kubernetes.io/projected/1196a70e-2a3b-46d1-8afa-2bf25e08fcb1-kube-api-access-lxmcx\") pod \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") "
Sep 18 17:19:19.590282 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:19.590231    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") pod \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\" (UID: \"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1\") "
Sep 18 17:19:19.592501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:19.592465    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1196a70e-2a3b-46d1-8afa-2bf25e08fcb1-kube-api-access-lxmcx" (OuterVolumeSpecName: "kube-api-access-lxmcx") pod "1196a70e-2a3b-46d1-8afa-2bf25e08fcb1" (UID: "1196a70e-2a3b-46d1-8afa-2bf25e08fcb1"). InnerVolumeSpecName "kube-api-access-lxmcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:19:19.594696 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:19.594668    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266" (OuterVolumeSpecName: "model-storage") pod "1196a70e-2a3b-46d1-8afa-2bf25e08fcb1" (UID: "1196a70e-2a3b-46d1-8afa-2bf25e08fcb1"). InnerVolumeSpecName "pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 17:19:19.691053 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:19.690969    2562 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") on node \"ip-10-0-130-101.ec2.internal\" "
Sep 18 17:19:19.691053 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:19.690999    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmcx\" (UniqueName: \"kubernetes.io/projected/1196a70e-2a3b-46d1-8afa-2bf25e08fcb1-kube-api-access-lxmcx\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:19:20.422181 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:20.422147    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"
Sep 18 17:19:20.422392 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:20.422187    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq" event={"ID":"1196a70e-2a3b-46d1-8afa-2bf25e08fcb1","Type":"ContainerDied","Data":"9fbacac5a1cc7d326a68f33e3655c3909458f194a3d63638a9f31c6ab99b80c2"}
Sep 18 17:19:20.422392 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:20.422230    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fbacac5a1cc7d326a68f33e3655c3909458f194a3d63638a9f31c6ab99b80c2"
Sep 18 17:19:20.660142 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:20.660111    2562 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266") on node "ip-10-0-130-101.ec2.internal"
Sep 18 17:19:20.700759 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:20.700719    2562 reconciler_common.go:299] "Volume detached for volume \"pvc-8b8c8904-89ff-4e63-bbe0-4abeeecc0063\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0221ad248af1f2266\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"csi-2ec842590097daf87e68d99522cb2f514630e473ed082d3808c0ec85a9a9a54e\""
Sep 18 17:19:25.683615 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:25.683574    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:19:35.683588 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:35.683538    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:19:45.683725 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:45.683675    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:19:55.683347 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:19:55.683297    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:20:05.683185 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:05.683136    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": dial tcp 10.134.0.55:8000: connect: connection refused"
Sep 18 17:20:15.693612 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:15.693577    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:20:15.701730 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:15.701694    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:20:25.135568 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:25.135527    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"]
Sep 18 17:20:40.161006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:40.160942    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main" containerID="cri-o://abc96e1ef7733eeb791eae59814176c04c703f64f1fe80d1054b38b7fec7b3fb" gracePeriod=45
Sep 18 17:20:40.886076 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:40.886040    2562 generic.go:356] "Generic (PLEG): container finished" podID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerID="abc96e1ef7733eeb791eae59814176c04c703f64f1fe80d1054b38b7fec7b3fb" exitCode=0
Sep 18 17:20:40.886290 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:40.886102    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" event={"ID":"2a285ac7-0fa6-42af-93de-48a0297423e8","Type":"ContainerDied","Data":"abc96e1ef7733eeb791eae59814176c04c703f64f1fe80d1054b38b7fec7b3fb"}
Sep 18 17:20:40.917703 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:40.917677    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:20:41.015645 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015553    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-home\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.015815 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015656    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j569\" (UniqueName: \"kubernetes.io/projected/2a285ac7-0fa6-42af-93de-48a0297423e8-kube-api-access-2j569\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.015815 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015702    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-kserve-provision-location\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.015815 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015745    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a285ac7-0fa6-42af-93de-48a0297423e8-cabundle-cert\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.015815 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-model-cache\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.016529 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015826    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-tmp-dir\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.016529 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015860    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-dshm\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.016529 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.015926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a285ac7-0fa6-42af-93de-48a0297423e8-tls-certs\") pod \"2a285ac7-0fa6-42af-93de-48a0297423e8\" (UID: \"2a285ac7-0fa6-42af-93de-48a0297423e8\") "
Sep 18 17:20:41.016529 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.016087    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.016529 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.016217    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.016529 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.016374    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-home" (OuterVolumeSpecName: "home") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.016827 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.016659    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a285ac7-0fa6-42af-93de-48a0297423e8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:20:41.018283 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.018255    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a285ac7-0fa6-42af-93de-48a0297423e8-kube-api-access-2j569" (OuterVolumeSpecName: "kube-api-access-2j569") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "kube-api-access-2j569". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:20:41.018416 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.018333    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a285ac7-0fa6-42af-93de-48a0297423e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:20:41.018662 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.018627    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-dshm" (OuterVolumeSpecName: "dshm") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.029869 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.029826    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.043457 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.043407    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2a285ac7-0fa6-42af-93de-48a0297423e8" (UID: "2a285ac7-0fa6-42af-93de-48a0297423e8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:20:41.117386 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.117345    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a285ac7-0fa6-42af-93de-48a0297423e8-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.117386 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.117376    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.117386 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.117386    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j569\" (UniqueName: \"kubernetes.io/projected/2a285ac7-0fa6-42af-93de-48a0297423e8-kube-api-access-2j569\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.117386 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.117395    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.117670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.117406    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2a285ac7-0fa6-42af-93de-48a0297423e8-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.117670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.117414    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.117670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.117421    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a285ac7-0fa6-42af-93de-48a0297423e8-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:20:41.893099 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.893060    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj" event={"ID":"2a285ac7-0fa6-42af-93de-48a0297423e8","Type":"ContainerDied","Data":"aaf2bd35a61a5fdcbffec97804523fbc75cdefb04d1a1520bcccf609dfa9273b"}
Sep 18 17:20:41.893631 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.893103    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"
Sep 18 17:20:41.893631 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.893127    2562 scope.go:117] "RemoveContainer" containerID="abc96e1ef7733eeb791eae59814176c04c703f64f1fe80d1054b38b7fec7b3fb"
Sep 18 17:20:41.903283 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.903260    2562 scope.go:117] "RemoveContainer" containerID="0c0dea64052be737dfdd82d8050f5949a734099ef1188e8e4856bddda171bfef"
Sep 18 17:20:41.917575 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.917546    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"]
Sep 18 17:20:41.921562 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:41.921535    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-7b7d6f8c6b-m4nqj"]
Sep 18 17:20:42.335694 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:20:42.335613    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" path="/var/lib/kubelet/pods/2a285ac7-0fa6-42af-93de-48a0297423e8/volumes"
Sep 18 17:21:15.454410 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.454346    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"]
Sep 18 17:21:15.455013 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.454963    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="storage-initializer"
Sep 18 17:21:15.455013 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.454990    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="storage-initializer"
Sep 18 17:21:15.455123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.455015    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main"
Sep 18 17:21:15.455123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.455024    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main"
Sep 18 17:21:15.455123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.455059    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1196a70e-2a3b-46d1-8afa-2bf25e08fcb1" containerName="storage-initializer"
Sep 18 17:21:15.455123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.455068    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1196a70e-2a3b-46d1-8afa-2bf25e08fcb1" containerName="storage-initializer"
Sep 18 17:21:15.455304 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.455159    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1196a70e-2a3b-46d1-8afa-2bf25e08fcb1" containerName="storage-initializer"
Sep 18 17:21:15.455304 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.455178    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a285ac7-0fa6-42af-93de-48a0297423e8" containerName="main"
Sep 18 17:21:15.458828 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.458804    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.461899 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.461873    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Sep 18 17:21:15.462069 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.462049    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-pvc6f\""
Sep 18 17:21:15.462312 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.462293    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 17:21:15.462421 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.462335    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 17:21:15.471108 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.471078    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"]
Sep 18 17:21:15.508746 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.508706    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.508746 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.508750    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.508970 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.508819    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.508970 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.508895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.508970 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.508924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.508970 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.508950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dt6\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-kube-api-access-64dt6\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.509101 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.508988    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.509101 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.509017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.509101 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.509032    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610166 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610111    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610166 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610160    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dt6\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-kube-api-access-64dt6\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610205    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610279    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610299    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610346    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610434 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610803 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610775    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610862 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-credential-socket\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610862 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610852    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-data\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.610970 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.610934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-certs\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.611263 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.611219    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.612842 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.612814    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-envoy\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.613057 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.613038    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-podinfo\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.618435 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.618402    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-token\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.618435 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.618422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dt6\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-kube-api-access-64dt6\") pod \"router-gateway-2-openshift-default-75b4cf4b5b-vmdv9\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:15.771125 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:15.771018    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:16.053106 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:16.053066    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"]
Sep 18 17:21:16.055200 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:21:16.055172    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6200b1_c1ba_42c9_a5b8_41f84d362650.slice/crio-fbce11a55480b155a4615e91dbb9d043dc332bf8d4f9fd1c7a639f588d461db2 WatchSource:0}: Error finding container fbce11a55480b155a4615e91dbb9d043dc332bf8d4f9fd1c7a639f588d461db2: Status 404 returned error can't find the container with id fbce11a55480b155a4615e91dbb9d043dc332bf8d4f9fd1c7a639f588d461db2
Sep 18 17:21:16.057726 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:16.057692    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:21:16.057815 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:16.057760    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:21:16.057815 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:16.057789    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:21:16.082334 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:16.082295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" event={"ID":"7c6200b1-c1ba-42c9-a5b8-41f84d362650","Type":"ContainerStarted","Data":"fbce11a55480b155a4615e91dbb9d043dc332bf8d4f9fd1c7a639f588d461db2"}
Sep 18 17:21:17.090386 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:17.090345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" event={"ID":"7c6200b1-c1ba-42c9-a5b8-41f84d362650","Type":"ContainerStarted","Data":"d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba"}
Sep 18 17:21:17.118077 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:17.118007    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podStartSLOduration=2.117982903 podStartE2EDuration="2.117982903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:21:17.117269142 +0000 UTC m=+3910.892701323" watchObservedRunningTime="2026-09-18 17:21:17.117982903 +0000 UTC m=+3910.893415066"
Sep 18 17:21:17.771672 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:17.771574    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:17.777130 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:17.777096    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:18.096324 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:18.096206    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:18.138432 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:18.138394    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:27.147740 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:27.147703    2562 generic.go:356] "Generic (PLEG): container finished" podID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerID="d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba" exitCode=0
Sep 18 17:21:27.148198 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:27.147780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" event={"ID":"7c6200b1-c1ba-42c9-a5b8-41f84d362650","Type":"ContainerDied","Data":"d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba"}
Sep 18 17:21:27.148198 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:27.148191    2562 scope.go:117] "RemoveContainer" containerID="d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba"
Sep 18 17:21:27.149161 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:27.149135    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:21:27.149253 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:27.149225    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:21:27.149303 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:27.149285    2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 18 17:21:28.164012 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:28.163967    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" event={"ID":"7c6200b1-c1ba-42c9-a5b8-41f84d362650","Type":"ContainerStarted","Data":"edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4"}
Sep 18 17:21:28.772089 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:28.772041    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:28.773329 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:28.773293    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 17:21:29.771788 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:29.771738    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 17:21:30.748577 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.748540    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"]
Sep 18 17:21:30.752168 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.752140    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.757594 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.757563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 17:21:30.757822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.757563    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 17:21:30.758032 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.757652    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:21:30.758181 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.757768    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-v5kh6\""
Sep 18 17:21:30.771003 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.770954    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"]
Sep 18 17:21:30.772123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.771408    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 17:21:30.866828 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.866776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-home\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.867047 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.866879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/124de5e2-5f55-47a9-bd96-663c126243b9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.867047 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.866930    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.867047 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.866948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.867047 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.866979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124de5e2-5f55-47a9-bd96-663c126243b9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.867321 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.867084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.867321 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.867124    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67msv\" (UniqueName: \"kubernetes.io/projected/124de5e2-5f55-47a9-bd96-663c126243b9-kube-api-access-67msv\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.867321 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.867159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.967939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.967896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/124de5e2-5f55-47a9-bd96-663c126243b9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.967939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.967944    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968147 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.967964    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968147 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124de5e2-5f55-47a9-bd96-663c126243b9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968213 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968287 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968218    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67msv\" (UniqueName: \"kubernetes.io/projected/124de5e2-5f55-47a9-bd96-663c126243b9-kube-api-access-67msv\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968328 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968383 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968366    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-home\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968427 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968399    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968720 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968695    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968836 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968730    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968836 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-home\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.968988 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.968959    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/124de5e2-5f55-47a9-bd96-663c126243b9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.970753 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.970733    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.970909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.970892    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124de5e2-5f55-47a9-bd96-663c126243b9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:30.976720 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:30.976696    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67msv\" (UniqueName: \"kubernetes.io/projected/124de5e2-5f55-47a9-bd96-663c126243b9-kube-api-access-67msv\") pod \"router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:31.088295 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:31.088170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:31.397820 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:31.397782    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"]
Sep 18 17:21:31.400999 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:21:31.400970    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod124de5e2_5f55_47a9_bd96_663c126243b9.slice/crio-6493d64457bd62204723e2c722949751b533f0a43f8078f9cf13adec67e0e0eb WatchSource:0}: Error finding container 6493d64457bd62204723e2c722949751b533f0a43f8078f9cf13adec67e0e0eb: Status 404 returned error can't find the container with id 6493d64457bd62204723e2c722949751b533f0a43f8078f9cf13adec67e0e0eb
Sep 18 17:21:31.771699 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:31.771587    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 17:21:32.194115 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:32.194069    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" event={"ID":"124de5e2-5f55-47a9-bd96-663c126243b9","Type":"ContainerStarted","Data":"024a98e4c096c9f9a5ece173586b16f83cf79eb30c87e6c55560a71034f6a4df"}
Sep 18 17:21:32.194520 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:32.194142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" event={"ID":"124de5e2-5f55-47a9-bd96-663c126243b9","Type":"ContainerStarted","Data":"6493d64457bd62204723e2c722949751b533f0a43f8078f9cf13adec67e0e0eb"}
Sep 18 17:21:32.772956 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:32.772547    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Sep 18 17:21:33.096888 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:33.096757    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:33.776482 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:33.776449    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:33.778056 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:33.778025    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:21:34.207187 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:34.207152    2562 generic.go:356] "Generic (PLEG): container finished" podID="124de5e2-5f55-47a9-bd96-663c126243b9" containerID="024a98e4c096c9f9a5ece173586b16f83cf79eb30c87e6c55560a71034f6a4df" exitCode=0
Sep 18 17:21:34.207385 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:34.207224    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" event={"ID":"124de5e2-5f55-47a9-bd96-663c126243b9","Type":"ContainerDied","Data":"024a98e4c096c9f9a5ece173586b16f83cf79eb30c87e6c55560a71034f6a4df"}
Sep 18 17:21:35.216471 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:35.216437    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" event={"ID":"124de5e2-5f55-47a9-bd96-663c126243b9","Type":"ContainerStarted","Data":"f2600f168c453b3c9521a1a23821d3dba4e98179a978f04444de0b76ec85ceed"}
Sep 18 17:21:35.240776 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:35.240716    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podStartSLOduration=2.617962323 podStartE2EDuration="5.240699465s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.622737142s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:21:35.238038977 +0000 UTC m=+3929.013471158" watchObservedRunningTime="2026-09-18 17:21:35.240699465 +0000 UTC m=+3929.016131626"
Sep 18 17:21:41.088865 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:41.088812    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:41.089301 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:41.088909    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:21:41.090660 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:41.090629    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:21:51.089231 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:21:51.089141    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:22:01.088646 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:01.088591    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:22:04.926815 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:04.926767    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"]
Sep 18 17:22:04.930279 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:04.930246    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/e2e-pvc-lora-download-nnvkq"]
Sep 18 17:22:06.334814 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:06.334776    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1196a70e-2a3b-46d1-8afa-2bf25e08fcb1" path="/var/lib/kubelet/pods/1196a70e-2a3b-46d1-8afa-2bf25e08fcb1/volumes"
Sep 18 17:22:11.089059 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:11.089004    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:22:21.088817 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:21.088760    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:22:31.088808 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:31.088758    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:22:41.089375 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:41.089319    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:22:51.088906 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:22:51.088856    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:23:01.089018 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:23:01.088965    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:23:11.089535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:23:11.089482    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:23:21.095552 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:23:21.095445    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:23:31.089310 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:23:31.089220    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": dial tcp 10.134.0.58:8000: connect: connection refused"
Sep 18 17:23:41.099270 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:23:41.099215    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:23:41.107385 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:23:41.107352    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:23:50.441920 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:23:50.441876    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"]
Sep 18 17:24:05.466519 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:05.466470    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main" containerID="cri-o://f2600f168c453b3c9521a1a23821d3dba4e98179a978f04444de0b76ec85ceed" gracePeriod=45
Sep 18 17:24:06.087402 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.087357    2562 generic.go:356] "Generic (PLEG): container finished" podID="124de5e2-5f55-47a9-bd96-663c126243b9" containerID="f2600f168c453b3c9521a1a23821d3dba4e98179a978f04444de0b76ec85ceed" exitCode=0
Sep 18 17:24:06.087566 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.087408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" event={"ID":"124de5e2-5f55-47a9-bd96-663c126243b9","Type":"ContainerDied","Data":"f2600f168c453b3c9521a1a23821d3dba4e98179a978f04444de0b76ec85ceed"}
Sep 18 17:24:06.231431 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.231404    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:24:06.287897 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.287798    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-kserve-provision-location\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288061 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.287916    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-home\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288061 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.287945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-dshm\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288061 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.287989    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124de5e2-5f55-47a9-bd96-663c126243b9-tls-certs\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288061 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.288008    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67msv\" (UniqueName: \"kubernetes.io/projected/124de5e2-5f55-47a9-bd96-663c126243b9-kube-api-access-67msv\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288061 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.288037    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-tmp-dir\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288383 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.288084    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/124de5e2-5f55-47a9-bd96-663c126243b9-cabundle-cert\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288383 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.288111    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-model-cache\") pod \"124de5e2-5f55-47a9-bd96-663c126243b9\" (UID: \"124de5e2-5f55-47a9-bd96-663c126243b9\") "
Sep 18 17:24:06.288676 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.288633    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.288835 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.288560    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/124de5e2-5f55-47a9-bd96-663c126243b9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:24:06.289257 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.289210    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-home" (OuterVolumeSpecName: "home") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.290549 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.290518    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-dshm" (OuterVolumeSpecName: "dshm") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.290664 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.290583    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124de5e2-5f55-47a9-bd96-663c126243b9-kube-api-access-67msv" (OuterVolumeSpecName: "kube-api-access-67msv") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "kube-api-access-67msv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:24:06.290728 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.290668    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124de5e2-5f55-47a9-bd96-663c126243b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:24:06.300784 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.300748    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.314024 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.313970    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "124de5e2-5f55-47a9-bd96-663c126243b9" (UID: "124de5e2-5f55-47a9-bd96-663c126243b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:06.389820 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389780    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/124de5e2-5f55-47a9-bd96-663c126243b9-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.389820 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389822    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.390035 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389836    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.390035 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389845    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.390035 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389855    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.390035 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389863    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124de5e2-5f55-47a9-bd96-663c126243b9-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.390035 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389872    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67msv\" (UniqueName: \"kubernetes.io/projected/124de5e2-5f55-47a9-bd96-663c126243b9-kube-api-access-67msv\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:06.390035 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:06.389879    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124de5e2-5f55-47a9-bd96-663c126243b9-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:07.095665 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:07.095626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m" event={"ID":"124de5e2-5f55-47a9-bd96-663c126243b9","Type":"ContainerDied","Data":"6493d64457bd62204723e2c722949751b533f0a43f8078f9cf13adec67e0e0eb"}
Sep 18 17:24:07.095665 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:07.095666    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"
Sep 18 17:24:07.096219 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:07.095699    2562 scope.go:117] "RemoveContainer" containerID="f2600f168c453b3c9521a1a23821d3dba4e98179a978f04444de0b76ec85ceed"
Sep 18 17:24:07.105051 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:07.105028    2562 scope.go:117] "RemoveContainer" containerID="024a98e4c096c9f9a5ece173586b16f83cf79eb30c87e6c55560a71034f6a4df"
Sep 18 17:24:07.117734 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:07.117707    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"]
Sep 18 17:24:07.121406 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:07.121383    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-669868b968-5pw6m"]
Sep 18 17:24:08.336276 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:08.336227    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" path="/var/lib/kubelet/pods/124de5e2-5f55-47a9-bd96-663c126243b9/volumes"
Sep 18 17:24:45.476253 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:45.476205    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"]
Sep 18 17:24:45.476856 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:45.476437    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy" containerID="cri-o://edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4" gracePeriod=30
Sep 18 17:24:48.779340 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:48.779282    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 17:24:50.740548 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.740482    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:24:50.772868 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.772831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-certs\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773053 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.772892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-credential-socket\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773053 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.772926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-data\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773053 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.772971    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64dt6\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-kube-api-access-64dt6\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773053 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773025    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istiod-ca-cert\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773306 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773143    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-podinfo\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773306 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773164    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:50.773306 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773191    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-token\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773306 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773246    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-envoy\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773306 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773284    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-socket\") pod \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\" (UID: \"7c6200b1-c1ba-42c9-a5b8-41f84d362650\") "
Sep 18 17:24:50.773649 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773400    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:50.773649 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773470    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:50.773754 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.773696    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:50.774177 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.774087    2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-credential-socket\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.774177 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.774120    2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-data\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.774177 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.774136    2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-socket\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.774177 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.774151    2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-workload-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.774522 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.774274    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:24:50.776209 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.776157    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-kube-api-access-64dt6" (OuterVolumeSpecName: "kube-api-access-64dt6") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "kube-api-access-64dt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:24:50.776411 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.776279    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:24:50.776795 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.776763    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 17:24:50.777331 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.777302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c6200b1-c1ba-42c9-a5b8-41f84d362650" (UID: "7c6200b1-c1ba-42c9-a5b8-41f84d362650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:24:50.874659 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.874618    2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istiod-ca-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.874659 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.874654    2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-podinfo\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.874659 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.874663    2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-token\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.874903 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.874671    2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6200b1-c1ba-42c9-a5b8-41f84d362650-istio-envoy\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:50.874903 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:50.874681    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64dt6\" (UniqueName: \"kubernetes.io/projected/7c6200b1-c1ba-42c9-a5b8-41f84d362650-kube-api-access-64dt6\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:24:51.340813 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.340772    2562 generic.go:356] "Generic (PLEG): container finished" podID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerID="edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4" exitCode=0
Sep 18 17:24:51.341006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.340828    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"
Sep 18 17:24:51.341006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.340861    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" event={"ID":"7c6200b1-c1ba-42c9-a5b8-41f84d362650","Type":"ContainerDied","Data":"edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4"}
Sep 18 17:24:51.341006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.340921    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9" event={"ID":"7c6200b1-c1ba-42c9-a5b8-41f84d362650","Type":"ContainerDied","Data":"fbce11a55480b155a4615e91dbb9d043dc332bf8d4f9fd1c7a639f588d461db2"}
Sep 18 17:24:51.341006 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.340945    2562 scope.go:117] "RemoveContainer" containerID="edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4"
Sep 18 17:24:51.351271 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.351246    2562 scope.go:117] "RemoveContainer" containerID="d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba"
Sep 18 17:24:51.360345 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.360322    2562 scope.go:117] "RemoveContainer" containerID="edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4"
Sep 18 17:24:51.360661 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:24:51.360645    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4\": container with ID starting with edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4 not found: ID does not exist" containerID="edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4"
Sep 18 17:24:51.360706 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.360669    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4"} err="failed to get container status \"edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4\": rpc error: code = NotFound desc = could not find container \"edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4\": container with ID starting with edb7757b2ac494671915278c74762244b8fb9b984ea91721207b765bb3d12ef4 not found: ID does not exist"
Sep 18 17:24:51.360706 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.360686    2562 scope.go:117] "RemoveContainer" containerID="d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba"
Sep 18 17:24:51.360972 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:24:51.360956    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba\": container with ID starting with d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba not found: ID does not exist" containerID="d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba"
Sep 18 17:24:51.361010 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.360978    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba"} err="failed to get container status \"d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba\": rpc error: code = NotFound desc = could not find container \"d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba\": container with ID starting with d26dc3e38f28efd43eb665a81514c6df8ae2ce1a83ffe07577a475250c24acba not found: ID does not exist"
Sep 18 17:24:51.366262 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.366217    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"]
Sep 18 17:24:51.369765 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:51.369733    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-75b4cf4b5b-vmdv9"]
Sep 18 17:24:52.334954 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:24:52.334916    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" path="/var/lib/kubelet/pods/7c6200b1-c1ba-42c9-a5b8-41f84d362650/volumes"
Sep 18 17:25:17.535617 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:17.535582    2562 scope.go:117] "RemoveContainer" containerID="2e960fa7bf8ca420f25a1109a6bb7b8a4e1bd5afbf8266b56286a58b72ea42a9"
Sep 18 17:25:34.590605 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.590564    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"]
Sep 18 17:25:34.591768 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591738    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="storage-initializer"
Sep 18 17:25:34.591870 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591772    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="storage-initializer"
Sep 18 17:25:34.591870 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591784    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main"
Sep 18 17:25:34.591870 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591795    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main"
Sep 18 17:25:34.591870 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591814    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy"
Sep 18 17:25:34.591870 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591823    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy"
Sep 18 17:25:34.591870 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591842    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy"
Sep 18 17:25:34.591870 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591851    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy"
Sep 18 17:25:34.592099 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.591994    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy"
Sep 18 17:25:34.592099 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.592010    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="124de5e2-5f55-47a9-bd96-663c126243b9" containerName="main"
Sep 18 17:25:34.592213 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.592197    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6200b1-c1ba-42c9-a5b8-41f84d362650" containerName="istio-proxy"
Sep 18 17:25:34.598214 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.598187    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.601133 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.601106    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 18 17:25:34.601343 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.601147    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:25:34.601343 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.601333    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 17:25:34.601490 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.601469    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 18 17:25:34.601562 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.601538    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-w58bg\""
Sep 18 17:25:34.601622 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.601565    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 18 17:25:34.607549 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.607483    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"]
Sep 18 17:25:34.744994 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.744950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tmp-dir\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.744994 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.744995    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-home\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.745276 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.745017    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-model-cache\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.745276 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.745091    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kserve-provision-location\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.745276 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.745132    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-cabundle-cert\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.745276 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.745166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-dshm\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.745276 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.745213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tls-certs\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.745450 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.745278    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpw2t\" (UniqueName: \"kubernetes.io/projected/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kube-api-access-lpw2t\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846154 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tmp-dir\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846154 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-home\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846154 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846142    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-model-cache\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846479 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846174    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kserve-provision-location\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846479 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846385    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-cabundle-cert\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846479 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-dshm\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846641 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846478    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tmp-dir\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846641 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tls-certs\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846641 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846487    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-model-cache\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846641 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846538    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpw2t\" (UniqueName: \"kubernetes.io/projected/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kube-api-access-lpw2t\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846641 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846541    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-home\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.846641 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.846618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kserve-provision-location\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.847127 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.847106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-cabundle-cert\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.848975 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.848937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-dshm\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.849566 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.849544    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tls-certs\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.855858 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.855823    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpw2t\" (UniqueName: \"kubernetes.io/projected/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kube-api-access-lpw2t\") pod \"tls-verification-test-kserve-65bc48d74c-jz79s\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:34.912472 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:34.912429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:35.197811 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:35.197775    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"]
Sep 18 17:25:35.199670 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:25:35.199644    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813cfcbb_1f0b_4623_9a36_0ee7d858dea1.slice/crio-0db64631f3d435cedb99d386a2b2daf591c73d4792f6b54fbcc39e321a157952 WatchSource:0}: Error finding container 0db64631f3d435cedb99d386a2b2daf591c73d4792f6b54fbcc39e321a157952: Status 404 returned error can't find the container with id 0db64631f3d435cedb99d386a2b2daf591c73d4792f6b54fbcc39e321a157952
Sep 18 17:25:35.588406 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:35.588300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" event={"ID":"813cfcbb-1f0b-4623-9a36-0ee7d858dea1","Type":"ContainerStarted","Data":"b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8"}
Sep 18 17:25:35.588406 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:35.588371    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" event={"ID":"813cfcbb-1f0b-4623-9a36-0ee7d858dea1","Type":"ContainerStarted","Data":"0db64631f3d435cedb99d386a2b2daf591c73d4792f6b54fbcc39e321a157952"}
Sep 18 17:25:38.607193 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:38.607157    2562 generic.go:356] "Generic (PLEG): container finished" podID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerID="b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8" exitCode=0
Sep 18 17:25:38.607570 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:38.607228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" event={"ID":"813cfcbb-1f0b-4623-9a36-0ee7d858dea1","Type":"ContainerDied","Data":"b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8"}
Sep 18 17:25:39.616575 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:39.616536    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" event={"ID":"813cfcbb-1f0b-4623-9a36-0ee7d858dea1","Type":"ContainerStarted","Data":"c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70"}
Sep 18 17:25:39.642863 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:39.642790    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podStartSLOduration=3.27856446 podStartE2EDuration="5.642766118s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.364201658s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:25:39.642469218 +0000 UTC m=+4173.417901426" watchObservedRunningTime="2026-09-18 17:25:39.642766118 +0000 UTC m=+4173.418198281"
Sep 18 17:25:39.913340 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:39.913250    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:44.912875 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:44.912831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:25:44.914476 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:44.914443    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:25:54.913325 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:25:54.913271    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:26:04.913012 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:26:04.912954    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:26:14.913129 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:26:14.913080    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:26:24.913466 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:26:24.913422    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:26:34.913350 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:26:34.913298    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:26:44.913669 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:26:44.913621    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:26:54.913367 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:26:54.913310    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:27:04.913212 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:04.913162    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:27:14.913650 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:14.913597    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:27:17.957249 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.957205    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 17:27:17.960882 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.960862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:17.963778 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.963739    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:27:17.963778 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.963766    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 17:27:17.963991 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.963896    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-g8sr7\""
Sep 18 17:27:17.963991 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.963980    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 17:27:17.965061 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.965039    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-679vs\""
Sep 18 17:27:17.965275 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.965258    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 17:27:17.965374 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.965318    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 17:27:17.971197 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:17.971171    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 17:27:18.069304 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.069494 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069371    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.069494 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069462    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.069577 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311341f6-0eb9-4a3d-bec6-0787683dd911-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.069616 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.069652 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.069652 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069636    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311341f6-0eb9-4a3d-bec6-0787683dd911-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.069718 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.069677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv4ns\" (UniqueName: \"kubernetes.io/projected/311341f6-0eb9-4a3d-bec6-0787683dd911-kube-api-access-kv4ns\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171018 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.170980    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171228 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171083    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171228 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171110    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311341f6-0eb9-4a3d-bec6-0787683dd911-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171228 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171133    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv4ns\" (UniqueName: \"kubernetes.io/projected/311341f6-0eb9-4a3d-bec6-0787683dd911-kube-api-access-kv4ns\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171228 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171162    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171472 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171363    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171472 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171472 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311341f6-0eb9-4a3d-bec6-0787683dd911-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171631 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171712 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171803    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.171891 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.171837    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.172202 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.172180    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311341f6-0eb9-4a3d-bec6-0787683dd911-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.173621 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.173596    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.173753 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.173734    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311341f6-0eb9-4a3d-bec6-0787683dd911-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.184455 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.184422    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv4ns\" (UniqueName: \"kubernetes.io/projected/311341f6-0eb9-4a3d-bec6-0787683dd911-kube-api-access-kv4ns\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.273747 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.273591    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:27:18.559940 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:18.559911    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 17:27:18.561790 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:27:18.561760    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311341f6_0eb9_4a3d_bec6_0787683dd911.slice/crio-58536c72fbd21fa7e59f966a600f1b70ee2ef07cdc76ea19ac58ee259e73a7ea WatchSource:0}: Error finding container 58536c72fbd21fa7e59f966a600f1b70ee2ef07cdc76ea19ac58ee259e73a7ea: Status 404 returned error can't find the container with id 58536c72fbd21fa7e59f966a600f1b70ee2ef07cdc76ea19ac58ee259e73a7ea
Sep 18 17:27:19.197682 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:19.197639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"311341f6-0eb9-4a3d-bec6-0787683dd911","Type":"ContainerStarted","Data":"8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c"}
Sep 18 17:27:19.198132 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:19.197701    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"311341f6-0eb9-4a3d-bec6-0787683dd911","Type":"ContainerStarted","Data":"58536c72fbd21fa7e59f966a600f1b70ee2ef07cdc76ea19ac58ee259e73a7ea"}
Sep 18 17:27:21.210795 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:21.210752    2562 generic.go:356] "Generic (PLEG): container finished" podID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerID="8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c" exitCode=0
Sep 18 17:27:21.211180 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:21.210827    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"311341f6-0eb9-4a3d-bec6-0787683dd911","Type":"ContainerDied","Data":"8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c"}
Sep 18 17:27:22.218962 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:22.218914    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"311341f6-0eb9-4a3d-bec6-0787683dd911","Type":"ContainerStarted","Data":"ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275"}
Sep 18 17:27:22.245094 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:22.245013    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.713781636 podStartE2EDuration="5.244986911s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.531205275s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:27:22.241345326 +0000 UTC m=+4276.016777517" watchObservedRunningTime="2026-09-18 17:27:22.244986911 +0000 UTC m=+4276.020419074"
Sep 18 17:27:24.913160 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:24.913098    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:27:34.913859 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:34.913805    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Sep 18 17:27:44.922939 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:44.922902    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:27:44.935741 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:44.935708    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:27:50.016834 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:27:50.016742    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"]
Sep 18 17:28:05.046223 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:05.046156    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main" containerID="cri-o://c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70" gracePeriod=45
Sep 18 17:28:05.822723 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:05.822696    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:28:06.020535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020496    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-cabundle-cert\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.020747 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020562    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-model-cache\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.020747 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020596    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tls-certs\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.020747 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020655    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-home\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.020747 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-dshm\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.020747 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpw2t\" (UniqueName: \"kubernetes.io/projected/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kube-api-access-lpw2t\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.021027 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020749    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tmp-dir\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.021027 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020812    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kserve-provision-location\") pod \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\" (UID: \"813cfcbb-1f0b-4623-9a36-0ee7d858dea1\") "
Sep 18 17:28:06.021027 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020912    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-model-cache" (OuterVolumeSpecName: "model-cache") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:06.021027 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.020970    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:28:06.021257 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.021117    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.021257 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.021136    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.021790 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.021757    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-home" (OuterVolumeSpecName: "home") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:06.023642 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.023605    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:28:06.023741 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.023676    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kube-api-access-lpw2t" (OuterVolumeSpecName: "kube-api-access-lpw2t") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "kube-api-access-lpw2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:28:06.023741 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.023690    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-dshm" (OuterVolumeSpecName: "dshm") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:06.034200 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.034144    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:06.056998 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.056946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "813cfcbb-1f0b-4623-9a36-0ee7d858dea1" (UID: "813cfcbb-1f0b-4623-9a36-0ee7d858dea1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:28:06.122188 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.122146    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.122188 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.122180    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.122188 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.122188    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.122188 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.122197    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpw2t\" (UniqueName: \"kubernetes.io/projected/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kube-api-access-lpw2t\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.122483 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.122209    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.122483 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.122219    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/813cfcbb-1f0b-4623-9a36-0ee7d858dea1-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:28:06.508470 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.508424    2562 generic.go:356] "Generic (PLEG): container finished" podID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerID="c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70" exitCode=0
Sep 18 17:28:06.508670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.508500    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"
Sep 18 17:28:06.508670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.508549    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" event={"ID":"813cfcbb-1f0b-4623-9a36-0ee7d858dea1","Type":"ContainerDied","Data":"c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70"}
Sep 18 17:28:06.508670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.508633    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s" event={"ID":"813cfcbb-1f0b-4623-9a36-0ee7d858dea1","Type":"ContainerDied","Data":"0db64631f3d435cedb99d386a2b2daf591c73d4792f6b54fbcc39e321a157952"}
Sep 18 17:28:06.508670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.508666    2562 scope.go:117] "RemoveContainer" containerID="c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70"
Sep 18 17:28:06.519334 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.519311    2562 scope.go:117] "RemoveContainer" containerID="b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8"
Sep 18 17:28:06.529620 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.529565    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"]
Sep 18 17:28:06.530077 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.530049    2562 scope.go:117] "RemoveContainer" containerID="c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70"
Sep 18 17:28:06.530672 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:28:06.530648    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70\": container with ID starting with c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70 not found: ID does not exist" containerID="c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70"
Sep 18 17:28:06.530793 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.530679    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70"} err="failed to get container status \"c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70\": rpc error: code = NotFound desc = could not find container \"c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70\": container with ID starting with c2177eb3340076036d60a0a9fd8f35e091e6d9e2eb77300649af231ba3a52f70 not found: ID does not exist"
Sep 18 17:28:06.530793 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.530697    2562 scope.go:117] "RemoveContainer" containerID="b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8"
Sep 18 17:28:06.531007 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:28:06.530988    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8\": container with ID starting with b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8 not found: ID does not exist" containerID="b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8"
Sep 18 17:28:06.531049 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.531011    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8"} err="failed to get container status \"b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8\": rpc error: code = NotFound desc = could not find container \"b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8\": container with ID starting with b7ae3e5197461bf286073eb77353824f8b3a7ecf4e9288db8f93949edd53d1f8 not found: ID does not exist"
Sep 18 17:28:06.532159 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:06.532137    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-65bc48d74c-jz79s"]
Sep 18 17:28:08.336936 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:08.336896    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" path="/var/lib/kubelet/pods/813cfcbb-1f0b-4623-9a36-0ee7d858dea1/volumes"
Sep 18 17:28:54.098472 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.098428    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"]
Sep 18 17:28:54.099034 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.099013    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="storage-initializer"
Sep 18 17:28:54.099080 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.099037    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="storage-initializer"
Sep 18 17:28:54.099080 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.099066    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main"
Sep 18 17:28:54.099080 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.099073    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main"
Sep 18 17:28:54.099193 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.099182    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="813cfcbb-1f0b-4623-9a36-0ee7d858dea1" containerName="main"
Sep 18 17:28:54.104163 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.104136    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.106813 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.106777    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 17:28:54.106976 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.106842    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 17:28:54.107127 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.107106    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 17:28:54.107798 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.107775    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 17:28:54.107897 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.107798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 17:28:54.107967 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.107776    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-sfd7z\""
Sep 18 17:28:54.114908 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.114879    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"]
Sep 18 17:28:54.241525 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a924e2c-77b6-4b6e-a212-896c2d8ea326-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.241525 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.241822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-dshm\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.241822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241656    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.241822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241705    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.241822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-home\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.241822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241772    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.241822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.241815    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7qtq\" (UniqueName: \"kubernetes.io/projected/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kube-api-access-q7qtq\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.342672 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.342629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.342855 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.342686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-home\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.342855 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.342734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.342855 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.342759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7qtq\" (UniqueName: \"kubernetes.io/projected/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kube-api-access-q7qtq\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.342855 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.342833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a924e2c-77b6-4b6e-a212-896c2d8ea326-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343076 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.342879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343076 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.343027    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-dshm\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343195 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.343087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343195 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.343107    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-home\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343195 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.343120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-model-cache\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343195 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.343187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tmp-dir\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343502 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.343481    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kserve-provision-location\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.343820 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.343799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a924e2c-77b6-4b6e-a212-896c2d8ea326-cabundle-cert\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.345345 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.345321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-dshm\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.345672 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.345652    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tls-certs\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.351355 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.351284    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7qtq\" (UniqueName: \"kubernetes.io/projected/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kube-api-access-q7qtq\") pod \"prestop-hook-test-kserve-d957cb699-2rc7r\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.416448 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.416400    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:28:54.712637 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.712593    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"]
Sep 18 17:28:54.793576 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:54.793526    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" event={"ID":"3a924e2c-77b6-4b6e-a212-896c2d8ea326","Type":"ContainerStarted","Data":"bf5659a72956da1ca5a0c6a27fb6bf8b7fd686e0df28fb0890d9f814d6b0aefa"}
Sep 18 17:28:55.802168 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:55.802128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" event={"ID":"3a924e2c-77b6-4b6e-a212-896c2d8ea326","Type":"ContainerStarted","Data":"49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a"}
Sep 18 17:28:57.817185 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:57.817151    2562 generic.go:356] "Generic (PLEG): container finished" podID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerID="49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a" exitCode=0
Sep 18 17:28:57.817721 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:57.817253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" event={"ID":"3a924e2c-77b6-4b6e-a212-896c2d8ea326","Type":"ContainerDied","Data":"49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a"}
Sep 18 17:28:58.824560 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:58.824519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" event={"ID":"3a924e2c-77b6-4b6e-a212-896c2d8ea326","Type":"ContainerStarted","Data":"0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81"}
Sep 18 17:28:58.848516 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:58.848420    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podStartSLOduration=2.407325746 podStartE2EDuration="4.848396276s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.44107053s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:28:58.845638918 +0000 UTC m=+4372.621071105" watchObservedRunningTime="2026-09-18 17:28:58.848396276 +0000 UTC m=+4372.623828454"
Sep 18 17:28:59.417496 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:28:59.417443    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:29:04.416975 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:04.416937    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:29:04.418724 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:04.418686    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:29:14.417798 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:14.417743    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:29:24.270342 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:24.270296    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 17:29:24.417832 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:24.417778    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:29:34.417771 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:34.417716    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:29:39.293386 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:39.293318    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerName="main" containerID="cri-o://ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275" gracePeriod=45
Sep 18 17:29:40.061544 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.061515    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:29:40.068565 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.068526    2562 generic.go:356] "Generic (PLEG): container finished" podID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerID="ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275" exitCode=0
Sep 18 17:29:40.068718 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.068587    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 17:29:40.068718 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.068606    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"311341f6-0eb9-4a3d-bec6-0787683dd911","Type":"ContainerDied","Data":"ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275"}
Sep 18 17:29:40.068718 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.068672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"311341f6-0eb9-4a3d-bec6-0787683dd911","Type":"ContainerDied","Data":"58536c72fbd21fa7e59f966a600f1b70ee2ef07cdc76ea19ac58ee259e73a7ea"}
Sep 18 17:29:40.068718 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.068706    2562 scope.go:117] "RemoveContainer" containerID="ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275"
Sep 18 17:29:40.078027 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.078004    2562 scope.go:117] "RemoveContainer" containerID="8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c"
Sep 18 17:29:40.089092 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.089065    2562 scope.go:117] "RemoveContainer" containerID="ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275"
Sep 18 17:29:40.089500 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:29:40.089475    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275\": container with ID starting with ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275 not found: ID does not exist" containerID="ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275"
Sep 18 17:29:40.089590 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.089510    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275"} err="failed to get container status \"ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275\": rpc error: code = NotFound desc = could not find container \"ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275\": container with ID starting with ef0e8195d90be43f82e30d8419132edbd47dd506040d9ea5f612a1fa8029d275 not found: ID does not exist"
Sep 18 17:29:40.089590 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.089533    2562 scope.go:117] "RemoveContainer" containerID="8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c"
Sep 18 17:29:40.089912 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:29:40.089887    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c\": container with ID starting with 8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c not found: ID does not exist" containerID="8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c"
Sep 18 17:29:40.089993 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.089922    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c"} err="failed to get container status \"8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c\": rpc error: code = NotFound desc = could not find container \"8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c\": container with ID starting with 8fff8f6b345a6545ce8a0a67e52e13b83d96ae7a9b9dc46a27127bded6cf429c not found: ID does not exist"
Sep 18 17:29:40.163399 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-tmp-dir\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.163399 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163404    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv4ns\" (UniqueName: \"kubernetes.io/projected/311341f6-0eb9-4a3d-bec6-0787683dd911-kube-api-access-kv4ns\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.163673 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163457    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311341f6-0eb9-4a3d-bec6-0787683dd911-tls-certs\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.163673 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163499    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-kserve-provision-location\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.163673 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163566    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-dshm\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.163673 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163595    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311341f6-0eb9-4a3d-bec6-0787683dd911-cabundle-cert\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.163673 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163653    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-model-cache\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.163969 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.163688    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-home\") pod \"311341f6-0eb9-4a3d-bec6-0787683dd911\" (UID: \"311341f6-0eb9-4a3d-bec6-0787683dd911\") "
Sep 18 17:29:40.164251 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.164111    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-model-cache" (OuterVolumeSpecName: "model-cache") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:40.164366 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.164325    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311341f6-0eb9-4a3d-bec6-0787683dd911-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:29:40.164650 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.164592    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-home" (OuterVolumeSpecName: "home") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:40.166530 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.166498    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-dshm" (OuterVolumeSpecName: "dshm") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:40.166633 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.166611    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311341f6-0eb9-4a3d-bec6-0787683dd911-kube-api-access-kv4ns" (OuterVolumeSpecName: "kube-api-access-kv4ns") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "kube-api-access-kv4ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:29:40.166843 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.166820    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311341f6-0eb9-4a3d-bec6-0787683dd911-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:29:40.181470 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.181429    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:40.189952 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.189911    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "311341f6-0eb9-4a3d-bec6-0787683dd911" (UID: "311341f6-0eb9-4a3d-bec6-0787683dd911"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:29:40.265261 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265211    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.265261 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265269    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.265535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265281    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.265535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265292    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv4ns\" (UniqueName: \"kubernetes.io/projected/311341f6-0eb9-4a3d-bec6-0787683dd911-kube-api-access-kv4ns\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.265535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265301    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/311341f6-0eb9-4a3d-bec6-0787683dd911-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.265535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265309    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.265535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265317    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/311341f6-0eb9-4a3d-bec6-0787683dd911-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.265535 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.265328    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/311341f6-0eb9-4a3d-bec6-0787683dd911-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:29:40.386780 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.386737    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 17:29:40.389796 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:40.389768    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 17:29:42.336456 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:42.336420    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311341f6-0eb9-4a3d-bec6-0787683dd911" path="/var/lib/kubelet/pods/311341f6-0eb9-4a3d-bec6-0787683dd911/volumes"
Sep 18 17:29:44.417399 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:44.417342    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:29:54.417333 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:29:54.417285    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:30:04.417512 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:30:04.417464    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:30:14.417467 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:30:14.417416    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:30:24.416885 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:30:24.416835    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:30:34.417525 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:30:34.417448    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:30:44.416964 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:30:44.416909    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:30:54.417797 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:30:54.417746    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Sep 18 17:31:04.427051 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:04.427016    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:31:04.435099 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:04.435067    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:31:05.683460 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:05.683423    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"]
Sep 18 17:31:20.706927 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:20.706879    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" containerID="cri-o://0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81" gracePeriod=45
Sep 18 17:31:21.463975 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.463944    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:31:21.556847 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.556755    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-home\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.556996 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.556862    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tls-certs\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.556996 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.556889    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kserve-provision-location\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.556996 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.556912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a924e2c-77b6-4b6e-a212-896c2d8ea326-cabundle-cert\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.556996 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.556951    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-dshm\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.556996 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.556976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tmp-dir\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.557293 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.557015    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-model-cache\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.557293 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.557043    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7qtq\" (UniqueName: \"kubernetes.io/projected/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kube-api-access-q7qtq\") pod \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\" (UID: \"3a924e2c-77b6-4b6e-a212-896c2d8ea326\") "
Sep 18 17:31:21.557582 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.557492    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a924e2c-77b6-4b6e-a212-896c2d8ea326-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 17:31:21.557582 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.557511    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-home" (OuterVolumeSpecName: "home") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:31:21.557582 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.557563    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:31:21.559444 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.559413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-dshm" (OuterVolumeSpecName: "dshm") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:31:21.559548 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.559467    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:31:21.559670 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.559647    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kube-api-access-q7qtq" (OuterVolumeSpecName: "kube-api-access-q7qtq") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "kube-api-access-q7qtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:31:21.570311 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.570223    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:31:21.580079 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.580046    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a924e2c-77b6-4b6e-a212-896c2d8ea326" (UID: "3a924e2c-77b6-4b6e-a212-896c2d8ea326"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:31:21.657909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657875    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.657909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657905    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kserve-provision-location\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.657909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657916    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a924e2c-77b6-4b6e-a212-896c2d8ea326-cabundle-cert\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.658183 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657925    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.658183 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657933    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.658183 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657940    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.658183 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657948    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7qtq\" (UniqueName: \"kubernetes.io/projected/3a924e2c-77b6-4b6e-a212-896c2d8ea326-kube-api-access-q7qtq\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.658183 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.657955    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a924e2c-77b6-4b6e-a212-896c2d8ea326-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:31:21.662537 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.662510    2562 generic.go:356] "Generic (PLEG): container finished" podID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerID="0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81" exitCode=0
Sep 18 17:31:21.662693 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.662575    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"
Sep 18 17:31:21.662693 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.662595    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" event={"ID":"3a924e2c-77b6-4b6e-a212-896c2d8ea326","Type":"ContainerDied","Data":"0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81"}
Sep 18 17:31:21.662816 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.662691    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" event={"ID":"3a924e2c-77b6-4b6e-a212-896c2d8ea326","Type":"ContainerDied","Data":"bf5659a72956da1ca5a0c6a27fb6bf8b7fd686e0df28fb0890d9f814d6b0aefa"}
Sep 18 17:31:21.662816 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.662726    2562 scope.go:117] "RemoveContainer" containerID="0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81"
Sep 18 17:31:21.673501 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.673481    2562 scope.go:117] "RemoveContainer" containerID="49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a"
Sep 18 17:31:21.682740 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.682720    2562 scope.go:117] "RemoveContainer" containerID="0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81"
Sep 18 17:31:21.683067 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:31:21.683044    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81\": container with ID starting with 0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81 not found: ID does not exist" containerID="0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81"
Sep 18 17:31:21.683140 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.683076    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81"} err="failed to get container status \"0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81\": rpc error: code = NotFound desc = could not find container \"0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81\": container with ID starting with 0d8dd931f4e7d22970f31b09caa93419ec9cac43a00718310fcedf5d77576f81 not found: ID does not exist"
Sep 18 17:31:21.683140 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.683096    2562 scope.go:117] "RemoveContainer" containerID="49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a"
Sep 18 17:31:21.683375 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:31:21.683358    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a\": container with ID starting with 49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a not found: ID does not exist" containerID="49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a"
Sep 18 17:31:21.683423 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.683381    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a"} err="failed to get container status \"49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a\": rpc error: code = NotFound desc = could not find container \"49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a\": container with ID starting with 49935b0e5857e041da87ab5ad3e3c6b48301198966d90ac5a11fb4abb6d2523a not found: ID does not exist"
Sep 18 17:31:21.687910 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.687875    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"]
Sep 18 17:31:21.691599 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:21.691567    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r"]
Sep 18 17:31:22.335463 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:22.335429    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" path="/var/lib/kubelet/pods/3a924e2c-77b6-4b6e-a212-896c2d8ea326/volumes"
Sep 18 17:31:22.428507 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:31:22.428446    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-d957cb699-2rc7r" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 17:33:03.372180 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372135    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"]
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372556    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerName="main"
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372573    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerName="main"
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372584    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="storage-initializer"
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372591    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="storage-initializer"
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372602    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main"
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372606    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main"
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372615    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerName="storage-initializer"
Sep 18 17:33:03.372678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372620    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerName="storage-initializer"
Sep 18 17:33:03.372932 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372698    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="311341f6-0eb9-4a3d-bec6-0787683dd911" containerName="main"
Sep 18 17:33:03.372932 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.372708    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a924e2c-77b6-4b6e-a212-896c2d8ea326" containerName="main"
Sep 18 17:33:03.375877 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.375845    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.378554 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.378518    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 17:33:03.379555 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.379533    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 17:33:03.379678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.379595    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-657px\""
Sep 18 17:33:03.379678 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.379596    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 17:33:03.383977 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.383898    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"]
Sep 18 17:33:03.494213 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.494166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d520da-a274-4199-8070-d7e16ec4bcb8-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.494213 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.494215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.494470 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.494294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.494470 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.494331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.494470 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.494374    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.494470 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.494414    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7mt\" (UniqueName: \"kubernetes.io/projected/c7d520da-a274-4199-8070-d7e16ec4bcb8-kube-api-access-vk7mt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595089 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595332 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595103    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595332 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595134    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7mt\" (UniqueName: \"kubernetes.io/projected/c7d520da-a274-4199-8070-d7e16ec4bcb8-kube-api-access-vk7mt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595332 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d520da-a274-4199-8070-d7e16ec4bcb8-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595332 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595332 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595217    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595607 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595585    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-model-cache\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595650 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595602    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-tmp-dir\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.595650 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.595631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-home\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.597578 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.597555    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-dshm\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.597772 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.597756    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d520da-a274-4199-8070-d7e16ec4bcb8-tls-certs\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.602909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.602890    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7mt\" (UniqueName: \"kubernetes.io/projected/c7d520da-a274-4199-8070-d7e16ec4bcb8-kube-api-access-vk7mt\") pod \"rolling-upgrade-test-kserve-6dd8dbf85-c2wx6\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.689223 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.689123    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:03.965654 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:03.965623    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"]
Sep 18 17:33:03.968347 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:33:03.968316    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d520da_a274_4199_8070_d7e16ec4bcb8.slice/crio-b9bf0e97c57bfcbec87ff4f51d14014f8554dd970ccb25048a0bcc321cc59a29 WatchSource:0}: Error finding container b9bf0e97c57bfcbec87ff4f51d14014f8554dd970ccb25048a0bcc321cc59a29: Status 404 returned error can't find the container with id b9bf0e97c57bfcbec87ff4f51d14014f8554dd970ccb25048a0bcc321cc59a29
Sep 18 17:33:04.228151 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:04.228053    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" event={"ID":"c7d520da-a274-4199-8070-d7e16ec4bcb8","Type":"ContainerStarted","Data":"16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f"}
Sep 18 17:33:04.228151 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:04.228106    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" event={"ID":"c7d520da-a274-4199-8070-d7e16ec4bcb8","Type":"ContainerStarted","Data":"b9bf0e97c57bfcbec87ff4f51d14014f8554dd970ccb25048a0bcc321cc59a29"}
Sep 18 17:33:04.246497 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:04.246442    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" podStartSLOduration=1.246422221 podStartE2EDuration="1.246422221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 17:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:33:04.244939262 +0000 UTC m=+4618.020371461" watchObservedRunningTime="2026-09-18 17:33:04.246422221 +0000 UTC m=+4618.021854414"
Sep 18 17:33:04.689826 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:04.689782    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:13.689583 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:13.689535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:13.702339 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:13.702309    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:14.290594 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:14.290562    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:28.324689 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.324639    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"]
Sep 18 17:33:28.336818 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:33:28.336763    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 17:33:28.336818 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:33:28.336818 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 17:33:28.336818 ip-10-0-130-101 kubenswrapper[2562]:  > containerID="16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f" cmd=["/bin/sleep","15"]
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.336841    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" message="" err=<
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]:  >
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:33:28.336862    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" podUID="c7d520da-a274-4199-8070-d7e16ec4bcb8" containerName="main" containerID="cri-o://16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f"
Sep 18 17:33:28.337154 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.336920    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" podUID="c7d520da-a274-4199-8070-d7e16ec4bcb8" containerName="main" containerID="cri-o://16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f" gracePeriod=60
Sep 18 17:33:28.608699 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.608665    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:28.719722 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.719684    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-dshm\") pod \"c7d520da-a274-4199-8070-d7e16ec4bcb8\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") "
Sep 18 17:33:28.719909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.719746    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk7mt\" (UniqueName: \"kubernetes.io/projected/c7d520da-a274-4199-8070-d7e16ec4bcb8-kube-api-access-vk7mt\") pod \"c7d520da-a274-4199-8070-d7e16ec4bcb8\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") "
Sep 18 17:33:28.719909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.719775    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-home\") pod \"c7d520da-a274-4199-8070-d7e16ec4bcb8\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") "
Sep 18 17:33:28.719909 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.719824    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-tmp-dir\") pod \"c7d520da-a274-4199-8070-d7e16ec4bcb8\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") "
Sep 18 17:33:28.720070 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720040    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d520da-a274-4199-8070-d7e16ec4bcb8-tls-certs\") pod \"c7d520da-a274-4199-8070-d7e16ec4bcb8\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") "
Sep 18 17:33:28.720197 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720086    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-model-cache\") pod \"c7d520da-a274-4199-8070-d7e16ec4bcb8\" (UID: \"c7d520da-a274-4199-8070-d7e16ec4bcb8\") "
Sep 18 17:33:28.720197 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720109    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c7d520da-a274-4199-8070-d7e16ec4bcb8" (UID: "c7d520da-a274-4199-8070-d7e16ec4bcb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:28.720197 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720124    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-home" (OuterVolumeSpecName: "home") pod "c7d520da-a274-4199-8070-d7e16ec4bcb8" (UID: "c7d520da-a274-4199-8070-d7e16ec4bcb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:28.720423 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720390    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c7d520da-a274-4199-8070-d7e16ec4bcb8" (UID: "c7d520da-a274-4199-8070-d7e16ec4bcb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:28.720570 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720507    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-model-cache\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:33:28.720570 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720522    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-home\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:33:28.720570 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.720549    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-tmp-dir\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:33:28.722333 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.722300    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-dshm" (OuterVolumeSpecName: "dshm") pod "c7d520da-a274-4199-8070-d7e16ec4bcb8" (UID: "c7d520da-a274-4199-8070-d7e16ec4bcb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 17:33:28.722441 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.722324    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d520da-a274-4199-8070-d7e16ec4bcb8-kube-api-access-vk7mt" (OuterVolumeSpecName: "kube-api-access-vk7mt") pod "c7d520da-a274-4199-8070-d7e16ec4bcb8" (UID: "c7d520da-a274-4199-8070-d7e16ec4bcb8"). InnerVolumeSpecName "kube-api-access-vk7mt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 17:33:28.722441 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.722306    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7d520da-a274-4199-8070-d7e16ec4bcb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c7d520da-a274-4199-8070-d7e16ec4bcb8" (UID: "c7d520da-a274-4199-8070-d7e16ec4bcb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 17:33:28.821915 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.821876    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7d520da-a274-4199-8070-d7e16ec4bcb8-dshm\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:33:28.821915 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.821909    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7mt\" (UniqueName: \"kubernetes.io/projected/c7d520da-a274-4199-8070-d7e16ec4bcb8-kube-api-access-vk7mt\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:33:28.821915 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:28.821922    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7d520da-a274-4199-8070-d7e16ec4bcb8-tls-certs\") on node \"ip-10-0-130-101.ec2.internal\" DevicePath \"\""
Sep 18 17:33:29.364283 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.364227    2562 generic.go:356] "Generic (PLEG): container finished" podID="c7d520da-a274-4199-8070-d7e16ec4bcb8" containerID="16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f" exitCode=0
Sep 18 17:33:29.364700 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.364298    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"
Sep 18 17:33:29.364700 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.364338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" event={"ID":"c7d520da-a274-4199-8070-d7e16ec4bcb8","Type":"ContainerDied","Data":"16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f"}
Sep 18 17:33:29.364700 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.364397    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6" event={"ID":"c7d520da-a274-4199-8070-d7e16ec4bcb8","Type":"ContainerDied","Data":"b9bf0e97c57bfcbec87ff4f51d14014f8554dd970ccb25048a0bcc321cc59a29"}
Sep 18 17:33:29.364700 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.364420    2562 scope.go:117] "RemoveContainer" containerID="16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f"
Sep 18 17:33:29.374506 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.374485    2562 scope.go:117] "RemoveContainer" containerID="16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f"
Sep 18 17:33:29.374823 ip-10-0-130-101 kubenswrapper[2562]: E0918 17:33:29.374805    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f\": container with ID starting with 16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f not found: ID does not exist" containerID="16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f"
Sep 18 17:33:29.374890 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.374833    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f"} err="failed to get container status \"16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f\": rpc error: code = NotFound desc = could not find container \"16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f\": container with ID starting with 16933df1e7ac0ef6a0e3dc25e706d1326639d8b69cc44d27a0fafb9518a6a04f not found: ID does not exist"
Sep 18 17:33:29.387054 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.387017    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"]
Sep 18 17:33:29.390323 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:29.390296    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-6dd8dbf85-c2wx6"]
Sep 18 17:33:30.334941 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:33:30.334904    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7d520da-a274-4199-8070-d7e16ec4bcb8" path="/var/lib/kubelet/pods/c7d520da-a274-4199-8070-d7e16ec4bcb8/volumes"
Sep 18 17:34:39.763476 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:39.763437    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r2wkt_0c9e11c8-f06a-4418-845e-7261062359ec/discovery/0.log"
Sep 18 17:34:39.808269 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:39.808222    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656b55b86d-nx575_bedebae7-23bb-4bb2-8550-b66910de6e6d/router/0.log"
Sep 18 17:34:40.607503 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:40.607465    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r2wkt_0c9e11c8-f06a-4418-845e-7261062359ec/discovery/0.log"
Sep 18 17:34:40.643973 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:40.643940    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656b55b86d-nx575_bedebae7-23bb-4bb2-8550-b66910de6e6d/router/0.log"
Sep 18 17:34:41.649358 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:41.649318    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-kpmzq_c1b524c7-f54c-467f-af5e-ce472c62c824/manager/0.log"
Sep 18 17:34:41.658429 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:41.658397    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fbm42_ff20ff04-e532-4e19-a755-62ed27751afa/limitador/0.log"
Sep 18 17:34:41.675070 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:41.675039    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-62kp7_72396fa1-0a6d-450f-ba7e-f44e6faf8f97/manager/0.log"
Sep 18 17:34:50.726389 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:50.726356    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mfqv2_0ddf424d-4391-4045-82cd-6e1bdc342467/global-pull-secret-syncer/0.log"
Sep 18 17:34:50.934769 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:50.934731    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sbb97_03205fd0-2fad-4887-9de5-1afa590637f2/konnectivity-agent/0.log"
Sep 18 17:34:50.961310 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:50.961280    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-101.ec2.internal_4627661586c3b25fe944f47481cab351/haproxy/0.log"
Sep 18 17:34:56.006404 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:56.006370    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-kpmzq_c1b524c7-f54c-467f-af5e-ce472c62c824/manager/0.log"
Sep 18 17:34:56.027488 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:56.027455    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-fbm42_ff20ff04-e532-4e19-a755-62ed27751afa/limitador/0.log"
Sep 18 17:34:56.058293 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:56.058256    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-62kp7_72396fa1-0a6d-450f-ba7e-f44e6faf8f97/manager/0.log"
Sep 18 17:34:57.502861 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:57.502831    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-r8zp5_04bc1f71-dacf-4cdf-b84f-de5d1519911d/kube-state-metrics/0.log"
Sep 18 17:34:57.518850 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:57.518814    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-r8zp5_04bc1f71-dacf-4cdf-b84f-de5d1519911d/kube-rbac-proxy-main/0.log"
Sep 18 17:34:57.540272 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:57.540223    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-84c885658c-r8zp5_04bc1f71-dacf-4cdf-b84f-de5d1519911d/kube-rbac-proxy-self/0.log"
Sep 18 17:34:57.589693 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:57.589661    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-wmgqp_0a7469ca-1aa3-4484-9ef4-0fd40e7174dc/monitoring-plugin/0.log"
Sep 18 17:34:57.715372 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:57.715339    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhlhc_9a69f4a3-abae-4da2-957f-c10b107872b9/node-exporter/0.log"
Sep 18 17:34:57.732993 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:57.732967    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhlhc_9a69f4a3-abae-4da2-957f-c10b107872b9/kube-rbac-proxy/0.log"
Sep 18 17:34:57.776075 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:57.775987    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fhlhc_9a69f4a3-abae-4da2-957f-c10b107872b9/init-textfile/0.log"
Sep 18 17:34:58.317578 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:58.317493    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-jrjjk_adeb27ae-270f-4d03-99b0-eeb07bc30b3f/prometheus-operator-admission-webhook/0.log"
Sep 18 17:34:58.445844 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:58.445813    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d9498446-pl6zs_6412340b-599d-4df9-80ab-51cebf8702fe/thanos-query/0.log"
Sep 18 17:34:58.468194 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:58.468165    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d9498446-pl6zs_6412340b-599d-4df9-80ab-51cebf8702fe/kube-rbac-proxy-web/0.log"
Sep 18 17:34:58.489018 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:58.488991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d9498446-pl6zs_6412340b-599d-4df9-80ab-51cebf8702fe/kube-rbac-proxy/0.log"
Sep 18 17:34:58.509489 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:58.509459    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d9498446-pl6zs_6412340b-599d-4df9-80ab-51cebf8702fe/prom-label-proxy/0.log"
Sep 18 17:34:58.542123 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:58.542090    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d9498446-pl6zs_6412340b-599d-4df9-80ab-51cebf8702fe/kube-rbac-proxy-rules/0.log"
Sep 18 17:34:58.563526 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:34:58.563495    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67d9498446-pl6zs_6412340b-599d-4df9-80ab-51cebf8702fe/kube-rbac-proxy-metrics/0.log"
Sep 18 17:35:00.824007 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.823970    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-tlpft_aa146f83-a717-4a52-a2e8-fe3c5ccb42d2/download-server/0.log"
Sep 18 17:35:00.857047 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.857005    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"]
Sep 18 17:35:00.857486 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.857466    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7d520da-a274-4199-8070-d7e16ec4bcb8" containerName="main"
Sep 18 17:35:00.857585 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.857490    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7d520da-a274-4199-8070-d7e16ec4bcb8" containerName="main"
Sep 18 17:35:00.857585 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.857573    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7d520da-a274-4199-8070-d7e16ec4bcb8" containerName="main"
Sep 18 17:35:00.860698 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.860671    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:00.863306 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.863282    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jhnn\"/\"openshift-service-ca.crt\""
Sep 18 17:35:00.863306 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.863297    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jhnn\"/\"kube-root-ca.crt\""
Sep 18 17:35:00.863453 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.863282    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4jhnn\"/\"default-dockercfg-mqxfb\""
Sep 18 17:35:00.871653 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.871626    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"]
Sep 18 17:35:00.966485 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.966444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-sys\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:00.966682 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.966494    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-podres\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:00.966682 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.966519    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-proc\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:00.966682 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.966567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpwk6\" (UniqueName: \"kubernetes.io/projected/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-kube-api-access-bpwk6\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:00.966682 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:00.966625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-lib-modules\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067577 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-sys\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067577 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-podres\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067620    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-proc\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpwk6\" (UniqueName: \"kubernetes.io/projected/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-kube-api-access-bpwk6\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067676    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-lib-modules\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067680    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-sys\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067725    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-proc\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.067822 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-lib-modules\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.068037 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.067827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-podres\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.076195 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.076120    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpwk6\" (UniqueName: \"kubernetes.io/projected/10fd60cc-edc2-4a28-a3ec-9ae3a315d48e-kube-api-access-bpwk6\") pod \"perf-node-gather-daemonset-cthqb\" (UID: \"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e\") " pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.172068 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.172028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.404350 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.404319    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-88s9f_f7b3b2cf-1efa-4c5b-9a1f-499dc21290fc/volume-data-source-validator/0.log"
Sep 18 17:35:01.450728 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.450698    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"]
Sep 18 17:35:01.452870 ip-10-0-130-101 kubenswrapper[2562]: W0918 17:35:01.452839    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod10fd60cc_edc2_4a28_a3ec_9ae3a315d48e.slice/crio-caa845ed687e920a99f3874bcf662a6bcbd5761b15144ee35cab287da4c92bc4 WatchSource:0}: Error finding container caa845ed687e920a99f3874bcf662a6bcbd5761b15144ee35cab287da4c92bc4: Status 404 returned error can't find the container with id caa845ed687e920a99f3874bcf662a6bcbd5761b15144ee35cab287da4c92bc4
Sep 18 17:35:01.856319 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.856178    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb" event={"ID":"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e","Type":"ContainerStarted","Data":"943eb3782e68f3db9a3b9292d4e8b11d48c513a9c5106a84ef6e23a2cf68b297"}
Sep 18 17:35:01.856319 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.856270    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb" event={"ID":"10fd60cc-edc2-4a28-a3ec-9ae3a315d48e","Type":"ContainerStarted","Data":"caa845ed687e920a99f3874bcf662a6bcbd5761b15144ee35cab287da4c92bc4"}
Sep 18 17:35:01.856319 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.856304    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:01.873559 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:01.873495    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb" podStartSLOduration=1.873476631 podStartE2EDuration="1.873476631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 17:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 17:35:01.871438658 +0000 UTC m=+4735.646870819" watchObservedRunningTime="2026-09-18 17:35:01.873476631 +0000 UTC m=+4735.648908794"
Sep 18 17:35:02.438699 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:02.438671    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-km6fd_a7d64b9b-0a6d-404b-9f0a-317263c530f4/dns/0.log"
Sep 18 17:35:02.477035 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:02.477004    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-km6fd_a7d64b9b-0a6d-404b-9f0a-317263c530f4/kube-rbac-proxy/0.log"
Sep 18 17:35:02.569936 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:02.569908    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pz2lq_ce80f8e8-94a6-4662-8531-c679fe377eef/dns-node-resolver/0.log"
Sep 18 17:35:03.136869 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:03.136837    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-csgxn_5e1d1ef2-8d2a-447b-a817-d4c67229a9db/node-ca/0.log"
Sep 18 17:35:04.226332 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:04.226291    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r2wkt_0c9e11c8-f06a-4418-845e-7261062359ec/discovery/0.log"
Sep 18 17:35:04.277160 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:04.277132    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656b55b86d-nx575_bedebae7-23bb-4bb2-8550-b66910de6e6d/router/0.log"
Sep 18 17:35:04.816766 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:04.816734    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-clq9q_4f877dfe-a591-4047-b728-942d3115b0ab/serve-healthcheck-canary/0.log"
Sep 18 17:35:05.298881 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:05.298841    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8prt9_2ad01b02-1165-4579-bfda-8fb5c6535ab7/kube-rbac-proxy/0.log"
Sep 18 17:35:05.320253 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:05.320205    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8prt9_2ad01b02-1165-4579-bfda-8fb5c6535ab7/exporter/0.log"
Sep 18 17:35:05.348937 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:05.348909    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8prt9_2ad01b02-1165-4579-bfda-8fb5c6535ab7/extractor/0.log"
Sep 18 17:35:07.871251 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:07.871204    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4jhnn/perf-node-gather-daemonset-cthqb"
Sep 18 17:35:08.464101 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:08.464048    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-g22cc_c5b6d717-261d-4ef2-9f4c-b0addfa22d97/openshift-lws-operator/0.log"
Sep 18 17:35:10.438801 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:10.438767    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r2wkt_0c9e11c8-f06a-4418-845e-7261062359ec/discovery/0.log"
Sep 18 17:35:10.490122 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:10.490091    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656b55b86d-nx575_bedebae7-23bb-4bb2-8550-b66910de6e6d/router/0.log"
Sep 18 17:35:11.760546 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:11.760516    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-r2wkt_0c9e11c8-f06a-4418-845e-7261062359ec/discovery/0.log"
Sep 18 17:35:11.799908 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:11.799864    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-656b55b86d-nx575_bedebae7-23bb-4bb2-8550-b66910de6e6d/router/0.log"
Sep 18 17:35:15.093510 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:15.093476    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-s2vjk_7bcd7861-01c5-4535-9a4b-bf6648d1ca52/migrator/0.log"
Sep 18 17:35:15.114174 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:15.114142    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-s2vjk_7bcd7861-01c5-4535-9a4b-bf6648d1ca52/graceful-termination/0.log"
Sep 18 17:35:15.458228 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:15.458187    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-l4lld_362c0f6a-10ae-45e0-bcc4-ecddfbea6a64/kube-storage-version-migrator-operator/1.log"
Sep 18 17:35:15.459808 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:15.459777    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-55889f7b47-l4lld_362c0f6a-10ae-45e0-bcc4-ecddfbea6a64/kube-storage-version-migrator-operator/0.log"
Sep 18 17:35:16.407648 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.407616    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wz42_ee46447e-bf78-431a-8433-2f1c5b570e78/kube-multus-additional-cni-plugins/0.log"
Sep 18 17:35:16.429023 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.428989    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wz42_ee46447e-bf78-431a-8433-2f1c5b570e78/egress-router-binary-copy/0.log"
Sep 18 17:35:16.449328 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.449299    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wz42_ee46447e-bf78-431a-8433-2f1c5b570e78/cni-plugins/0.log"
Sep 18 17:35:16.472548 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.472519    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wz42_ee46447e-bf78-431a-8433-2f1c5b570e78/bond-cni-plugin/0.log"
Sep 18 17:35:16.493066 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.493036    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wz42_ee46447e-bf78-431a-8433-2f1c5b570e78/routeoverride-cni/0.log"
Sep 18 17:35:16.515172 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.515142    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wz42_ee46447e-bf78-431a-8433-2f1c5b570e78/whereabouts-cni-bincopy/0.log"
Sep 18 17:35:16.535983 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.535951    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wz42_ee46447e-bf78-431a-8433-2f1c5b570e78/whereabouts-cni/0.log"
Sep 18 17:35:16.901811 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.901776    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ds28f_1242b928-afef-4603-b4b1-f9d3ad271442/kube-multus/0.log"
Sep 18 17:35:16.997097 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:16.997065    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8g777_c5d52384-daab-49e1-8414-893a899382b1/network-metrics-daemon/0.log"
Sep 18 17:35:17.017056 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:17.017017    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8g777_c5d52384-daab-49e1-8414-893a899382b1/kube-rbac-proxy/0.log"
Sep 18 17:35:18.173917 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.173871    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/ovn-controller/0.log"
Sep 18 17:35:18.189649 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.189620    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/ovn-acl-logging/0.log"
Sep 18 17:35:18.217884 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.217845    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/kube-rbac-proxy-node/0.log"
Sep 18 17:35:18.244121 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.244087    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 17:35:18.261617 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.261585    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/northd/0.log"
Sep 18 17:35:18.283671 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.283631    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/nbdb/0.log"
Sep 18 17:35:18.308018 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.307986    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/sbdb/0.log"
Sep 18 17:35:18.392227 ip-10-0-130-101 kubenswrapper[2562]: I0918 17:35:18.392194    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gn65t_df999121-c7c6-47d9-97f9-c4bc4b0fe062/ovnkube-controller/0.log"