Sep 17 23:08:41.213986 ip-10-0-132-155 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 23:08:41.213999 ip-10-0-132-155 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 23:08:41.214011 ip-10-0-132-155 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 23:08:41.214231 ip-10-0-132-155 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 23:08:51.445356 ip-10-0-132-155 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 23:08:51.445372 ip-10-0-132-155 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot cefdc1b5f2fc49dda251d485a38a195f --
Sep 17 23:10:38.789221 ip-10-0-132-155 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 23:10:39.220232 ip-10-0-132-155 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:39.220232 ip-10-0-132-155 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 23:10:39.220232 ip-10-0-132-155 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:39.220232 ip-10-0-132-155 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 23:10:39.220232 ip-10-0-132-155 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 23:10:39.222299 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.222212    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 23:10:39.226360 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226344    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.226360 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226360    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226364    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226367    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226371    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226374    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226377    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226380    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226383    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226386    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226389    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226392    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226394    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226397    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226401    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226405    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226409    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226411    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226414    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226417    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226419    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.226424 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226422    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226425    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226427    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226436    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226439    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226441    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226444    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226446    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226449    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226453    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226456    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226458    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226461    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226464    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226467    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226469    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226472    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226474    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226477    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226480    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.226911 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226482    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226485    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226488    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226490    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226493    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226495    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226498    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226500    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226503    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226505    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226508    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226511    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226514    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226517    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226520    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226523    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226525    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226528    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226531    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226534    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226537    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.227402 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226540    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226543    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226545    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226548    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226550    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226553    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226556    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226558    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226561    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226564    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226566    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226569    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226571    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226574    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226577    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226579    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226582    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226584    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226587    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.227978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226590    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226592    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226595    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226598    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226603    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226605    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226610    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226612    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226615    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226617    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226620    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226623    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226625    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226628    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226630    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226634    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226637    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.228436 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.226640    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.229273 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229260    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.229273 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229272    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229275    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229279    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229282    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229285    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229289    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229292    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229294    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229298    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229301    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229303    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229306    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229309    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229311    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229314    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229318    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229322    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229324    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229327    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.229335 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229330    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229333    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229335    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229338    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229341    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229344    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229347    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229350    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229353    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229356    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229359    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229361    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229364    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229366    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229369    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229372    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229374    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229377    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229379    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.229817 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229382    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229384    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229387    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229391    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229394    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229396    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229399    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229401    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229405    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229408    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229411    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229414    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229416    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229419    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229422    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229426    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229429    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229432    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.230304 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229434    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229438    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229442    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229445    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229448    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229451    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229454    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229456    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229459    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229463    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229465    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229468    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229471    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229473    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229476    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229478    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229481    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229483    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229486    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229489    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.230745 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229491    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229494    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229497    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229499    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229502    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229504    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229507    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229509    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229513    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229516    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229519    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229521    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229524    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229526    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229529    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229532    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229534    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229537    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229539    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229542    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.231258 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229544    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229547    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229957    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229962    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229965    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229968    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229971    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229974    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229976    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229979    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229985    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229988    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229990    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229993    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229996    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.229999    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230001    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230004    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230007    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.231746 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230009    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230011    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230014    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230018    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230020    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230023    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230026    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230028    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230031    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230033    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230036    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230039    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230042    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230045    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230047    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230050    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230052    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230055    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230058    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.232242 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230060    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230063    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230065    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230068    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230070    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230074    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230076    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230079    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230081    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230084    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230086    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230089    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230091    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230094    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230096    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230100    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230103    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230106    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230108    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230111    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.232699 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230113    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230116    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230119    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230122    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230124    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230127    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230129    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230132    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230134    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230137    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230140    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230142    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230144    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230147    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230150    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230152    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230155    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230157    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230160    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.233249 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230162    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230167    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230170    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230174    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230176    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230179    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230182    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230184    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230187    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230190    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230193    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230195    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230198    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230201    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230203    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230206    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230208    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230211    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230215    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.233710 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230218    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230221    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230224    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230227    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230229    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230303    2562 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230336    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230345    2562 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230349    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230354    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230357    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230361    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230365    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230370    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230373    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230376    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230379    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230382    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230385    2562 flags.go:64] FLAG: --cgroup-root=""
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230388    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230392    2562 flags.go:64] FLAG: --client-ca-file=""
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230395    2562 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230400    2562 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230404    2562 flags.go:64] FLAG: --cluster-domain=""
Sep 17 23:10:39.234179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230407    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230410    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230413    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230417    2562 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230420    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230424    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230427    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230430    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230433    2562 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230436    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230439    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230442    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230445    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230449    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230452    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230455    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230458    2562 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230461    2562 flags.go:64] FLAG: --enable-server="true"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230464    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230468    2562 flags.go:64] FLAG: --event-burst="100"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230471    2562 flags.go:64] FLAG: --event-qps="50"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230474    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230478    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230481    2562 flags.go:64] FLAG: --eviction-hard=""
Sep 17 23:10:39.234748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230485    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230488    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230491    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230494    2562 flags.go:64] FLAG: --eviction-soft=""
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230497    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230500    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230504    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230507    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230510    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230513    2562 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230516    2562 flags.go:64] FLAG: --feature-gates=""
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230520    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230523    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230526    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230529    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230532    2562 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230535    2562 flags.go:64] FLAG: --help="false"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230538    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230541    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230544    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230547    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230550    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230553    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230556    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 23:10:39.235337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230559    2562 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230562    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230565    2562 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230568    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230571    2562 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230574    2562 flags.go:64] FLAG: --kube-reserved=""
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230576    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230580    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230583    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230587    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230590    2562 flags.go:64] FLAG: --lock-file=""
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230593    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230596    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230599    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230606    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230609    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230611    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230614    2562 flags.go:64] FLAG: --logging-format="text"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230617    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230621    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230624    2562 flags.go:64] FLAG: --manifest-url=""
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230626    2562 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230631    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230634    2562 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230638    2562 flags.go:64] FLAG: --max-pods="110"
Sep 17 23:10:39.235969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230641    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230644    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230647    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230650    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230653    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230656    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230659    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230666    2562 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230669    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230671    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230675    2562 flags.go:64] FLAG: --pod-cidr=""
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230678    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230683    2562 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230686    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230689    2562 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230692    2562 flags.go:64] FLAG: --port="10250"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230695    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230700    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e11248856d699d35"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230703    2562 flags.go:64] FLAG: --qos-reserved=""
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230705    2562 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230709    2562 flags.go:64] FLAG: --register-node="true"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230713    2562 flags.go:64] FLAG: --register-with-taints=""
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230717    2562 flags.go:64] FLAG: --registry-burst="10"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230719    2562 flags.go:64] FLAG: --registry-qps="5"
Sep 17 23:10:39.236686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230722    2562 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230725    2562 flags.go:64] FLAG: --reserved-memory=""
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230729    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230732    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230735    2562 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230738    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230741    2562 flags.go:64] FLAG: --runonce="false"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230744    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230747    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230750    2562 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230753    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230756    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230759    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230762    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230765    2562 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230768    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230771    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230774    2562 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230777    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230780    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230782    2562 flags.go:64] FLAG: --system-cgroups=""
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230785    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230790    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230793    2562 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230797    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 23:10:39.237295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230803    2562 flags.go:64] FLAG: --tls-min-version=""
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230806    2562 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230808    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230811    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230816    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230819    2562 flags.go:64] FLAG: --v="2"
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230823    2562 flags.go:64] FLAG: --version="false"
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230827    2562 flags.go:64] FLAG: --vmodule=""
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230830    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.230834    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230971    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230975    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230978    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230982    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230986    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230990    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230993    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230995    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.230998    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231001    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231004    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.237912 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231006    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231009    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231013    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231016    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231019    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231022    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231025    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231028    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231031    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231033    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231036    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231040    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231043    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231046    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231048    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231052    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231055    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231058    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231060    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.238447 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231063    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231066    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231068    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231071    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231074    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231077    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231079    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231082    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231085    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231087    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231090    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231093    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231095    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231098    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231101    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231103    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231106    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231108    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231111    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231114    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.238930 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231117    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231119    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231122    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231125    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231129    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231132    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231135    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231137    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231141    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231144    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231147    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231149    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231152    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231155    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231157    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231160    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231162    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231165    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231167    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231170    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.239439 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231173    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231175    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231179    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231183    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231186    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231189    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231192    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231194    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231197    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231199    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231202    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231204    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231207    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231210    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231212    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231215    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231219    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231222    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231225    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231228    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.239978 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231232    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231234    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231237    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231240    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231243    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231245    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231248    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231251    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.231253    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.231259    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.238833    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.238847    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.239728    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.239736    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 23:10:39.240464 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239791    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239797    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239802    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239806    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239809    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239812    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239815    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239818    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239821    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239824    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239827    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239830    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239832    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239835    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239838    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239841    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239844    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239846    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239849    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239852    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.240826 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239854    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239857    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239860    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239862    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239865    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239868    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239870    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239873    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239875    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239880    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239883    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239885    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239889    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239893    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239907    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239912    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239915    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239918    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239921    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239924    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.241334 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239927    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239930    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239932    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239935    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239938    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239940    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239943    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239946    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239948    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239951    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239954    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239956    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239959    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239962    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239965    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239967    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239970    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239972    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239975    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239978    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.241821 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239980    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239984    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239992    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239995    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.239998    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240001    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240003    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240006    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240009    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240011    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240014    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240016    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240019    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240022    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240024    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240027    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240030    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240032    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240035    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.242422 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240039    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240042    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240045    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240048    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240051    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240054    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240057    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240060    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240062    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240065    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240067    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240070    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240072    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240075    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240079    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240082    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240085    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240088    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240090    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.243201 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240093    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.240098    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240208    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240213    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240216    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240219    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240222    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240225    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240228    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240231    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240234    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240237    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240239    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240242    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240245    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 23:10:39.243783 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240248    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240250    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240253    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240256    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240258    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240261    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240263    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240266    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240268    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240271    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240274    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240276    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240279    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240282    2562 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240285    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240288    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240291    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240294    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240296    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240299    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 23:10:39.244179 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240301    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240304    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240308    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240310    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240313    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240315    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240318    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240320    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240324    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240327    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240330    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240333    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240336    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240338    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240341    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240343    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240346    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240349    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240351    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 23:10:39.244660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240354    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240356    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240359    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240363    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240366    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240369    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240373    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240377    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240380    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240383    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240386    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240389    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240392    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240395    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240397    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240400    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240403    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240406    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 23:10:39.245147 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240409    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240412    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240414    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240417    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240420    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240423    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240425    2562 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240428    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240431    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240434    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240437    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240439    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240442    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240445    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240447    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240450    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240453    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240455    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240458    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240460    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 23:10:39.245607 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240463    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240466    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240469    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240472    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240475    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240478    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240481    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240483    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:39.240486    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.240491    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.240609    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.243169    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.244382    2562 server.go:1019] "Starting client certificate rotation"
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.244514    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:39.246146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.244569    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 23:10:39.247458 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.247443    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:39.248567 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.248550    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 23:10:39.257736 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.257724    2562 log.go:25] "Validated CRI v1 runtime API"
Sep 17 23:10:39.261314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.261297    2562 log.go:25] "Validated CRI v1 image API"
Sep 17 23:10:39.262155 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.262140    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 23:10:39.268396 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.268374    2562 fs.go:135] Filesystem UUIDs: map[0158b1b3-e4a9-4d60-8d56-38bfd456e65d:/dev/nvme0n1p3 232f2a32-b38f-4ae4-8988-7c0e635a27c6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 23:10:39.268456 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.268395    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 23:10:39.271351 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.271333    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:39.274363 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.274254    2562 manager.go:216] Machine: {Timestamp:2026-09-17 23:10:39.273022001 +0000 UTC m=+0.367549030 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101657 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e0b16e33bebaf625f4e1551e3080a SystemUUID:ec2e0b16-e33b-ebaf-625f-4e1551e3080a BootID:cefdc1b5-f2fc-49dd-a251-d485a38a195f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c8:df:e2:2f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c8:df:e2:2f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:95:10:5f:51:25 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 23:10:39.274363 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.274358    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 23:10:39.274477 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.274467    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 23:10:39.277794 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.277772    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 23:10:39.277944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.277798    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 23:10:39.278630 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.278620    2562 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 23:10:39.278660 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.278633    2562 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 23:10:39.278660 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.278640    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:39.278660 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.278654    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 23:10:39.278745 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.278721    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 23:10:39.282354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282343    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:39.282478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282469    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 23:10:39.282521 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282497    2562 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 23:10:39.282521 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282506    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 23:10:39.282521 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282521    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 23:10:39.282633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282530    2562 kubelet.go:398] "Adding apiserver pod source"
Sep 17 23:10:39.282633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282541    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 23:10:39.282865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282853    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:39.282924 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.282876    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 23:10:39.286685 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.285284    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 23:10:39.289146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.289130    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 23:10:39.289207 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.289162    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 23:10:39.290563 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290551    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290568    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290576    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290581    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290589    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290594    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290607    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290612    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290619    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290624    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 23:10:39.290631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290632    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 23:10:39.290884 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.290637    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 23:10:39.291507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.291486    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 23:10:39.291507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.291500    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 23:10:39.292696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.292666    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wdld"
Sep 17 23:10:39.295236 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.295213    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 23:10:39.295236 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.295214    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 23:10:39.295236 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.295241    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 23:10:39.295410 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.295262    2562 server.go:1268] "Started kubelet"
Sep 17 23:10:39.295410 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.295285    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 23:10:39.296148 ip-10-0-132-155 systemd[1]: Started Kubernetes Kubelet.
Sep 17 23:10:39.296354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.296210    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 23:10:39.296354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.296295    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 23:10:39.297952 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.297937    2562 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 23:10:39.298368 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.298352    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 23:10:39.300787 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.300770    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5wdld"
Sep 17 23:10:39.301365 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.301338    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 23:10:39.302784 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.301509    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-155.ec2.internal.18d63e842a748a04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-155.ec2.internal,UID:ip-10-0-132-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-155.ec2.internal,},FirstTimestamp:2026-09-17 23:10:39.295236612 +0000 UTC m=+0.389763638,LastTimestamp:2026-09-17 23:10:39.295236612 +0000 UTC m=+0.389763638,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-155.ec2.internal,}"
Sep 17 23:10:39.308230 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.308212    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:39.308938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.308882    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 23:10:39.309221 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.309203    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:39.309742 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.309728    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 23:10:39.309872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.309850    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 23:10:39.309976 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.309759    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 23:10:39.310076 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310060    2562 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 23:10:39.310076 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310072    2562 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 23:10:39.310274 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310259    2562 factory.go:55] Registering systemd factory
Sep 17 23:10:39.310326 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310278    2562 factory.go:223] Registration of the systemd container factory successfully
Sep 17 23:10:39.310593 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310579    2562 factory.go:153] Registering CRI-O factory
Sep 17 23:10:39.310659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310597    2562 factory.go:223] Registration of the crio container factory successfully
Sep 17 23:10:39.310659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310650    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 23:10:39.310770 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310669    2562 factory.go:103] Registering Raw factory
Sep 17 23:10:39.310770 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.310683    2562 manager.go:1196] Started watching for new ooms in manager
Sep 17 23:10:39.311305 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.311283    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 23:10:39.311362 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.311295    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 23:10:39.311712 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.311698    2562 manager.go:349] Starting recovery of all containers
Sep 17 23:10:39.322176 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.322157    2562 manager.go:354] Recovery completed
Sep 17 23:10:39.326321 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.326307    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.329052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.329034    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.329134 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.329067    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.329134 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.329082    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.329609 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.329595    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 23:10:39.329609 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.329607    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 23:10:39.329737 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.329624    2562 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 23:10:39.329737 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.329622    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.330392 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.330375    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.330452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.330406    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.330452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.330420    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.330798 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.330785    2562 policy_none.go:49] "None policy: Start"
Sep 17 23:10:39.330923 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.330804    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 23:10:39.330923 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.330817    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 23:10:39.331940 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.331929    2562 policy_none.go:47] "Start"
Sep 17 23:10:39.410012 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.409699    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:39.411954 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.411217    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 23:10:39.412373 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.412357    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 23:10:39.412416 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.412411    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 23:10:39.412455 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.412437    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 23:10:39.412555 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.412536    2562 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 17 23:10:39.414670 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.414651    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:39.474551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.474482    2562 manager.go:337] "Starting Device Plugin manager"
Sep 17 23:10:39.474551 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.474528    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 23:10:39.474551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.474541    2562 server.go:87] "Starting device plugin registration server"
Sep 17 23:10:39.474878 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.474866    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 23:10:39.474934 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.474878    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 23:10:39.475022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.475004    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 23:10:39.475112 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.475088    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 23:10:39.475112 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.475097    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 23:10:39.475706 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.475689    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 23:10:39.475783 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.475741    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:39.513197 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.513169    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal"]
Sep 17 23:10:39.513308 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.513238    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.514212 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.514187    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.514330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.514223    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.514330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.514238    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.516440 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.516427    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.516550 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.516536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.516595 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.516565    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.516951 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.516931    2562 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-155.ec2.internal\" not found" node="ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.517165 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.517147    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.517223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.517174    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.517223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.517184    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.517284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.517159    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.517284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.517241    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.517284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.517249    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.519179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.519164    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.519269 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.519187    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.519825 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.519809    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.519910 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.519834    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.519910 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.519844    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.533460 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.533437    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-155.ec2.internal\" not found" node="ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.537461 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.537446    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-155.ec2.internal\" not found" node="ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.575840 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.575811    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 23:10:39.576813 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.576797    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 23:10:39.576883 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.576828    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 23:10:39.576883 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.576840    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeHasSufficientPID"
Sep 17 23:10:39.576883 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.576867    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.587563 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.587542    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.587617 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.587568    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-155.ec2.internal\": node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:39.606183 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.606156    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:39.611430 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.611412    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bedd9a6fd6ace9132b55e90ebe8bf8b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal\" (UID: \"bedd9a6fd6ace9132b55e90ebe8bf8b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.611504 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.611438    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bedd9a6fd6ace9132b55e90ebe8bf8b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal\" (UID: \"bedd9a6fd6ace9132b55e90ebe8bf8b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.611504 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.611455    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52838b2473a80c0fab97ff455ee0daf5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-155.ec2.internal\" (UID: \"52838b2473a80c0fab97ff455ee0daf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.706776 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.706736    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:39.712101 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.712085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bedd9a6fd6ace9132b55e90ebe8bf8b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal\" (UID: \"bedd9a6fd6ace9132b55e90ebe8bf8b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.712153 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.712108    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bedd9a6fd6ace9132b55e90ebe8bf8b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal\" (UID: \"bedd9a6fd6ace9132b55e90ebe8bf8b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.712153 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.712127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52838b2473a80c0fab97ff455ee0daf5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-155.ec2.internal\" (UID: \"52838b2473a80c0fab97ff455ee0daf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.712219 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.712175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bedd9a6fd6ace9132b55e90ebe8bf8b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal\" (UID: \"bedd9a6fd6ace9132b55e90ebe8bf8b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.712219 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.712188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bedd9a6fd6ace9132b55e90ebe8bf8b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal\" (UID: \"bedd9a6fd6ace9132b55e90ebe8bf8b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.712281 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.712178    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/52838b2473a80c0fab97ff455ee0daf5-config\") pod \"kube-apiserver-proxy-ip-10-0-132-155.ec2.internal\" (UID: \"52838b2473a80c0fab97ff455ee0daf5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.807604 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.807520    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:39.837060 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.837028    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.840857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:39.840840    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:39.909671 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:39.909614    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:40.017268 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:40.017225    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:40.117372 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:40.117290    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:40.125162 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.125131    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:40.217558 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:40.217516    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:40.230208 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.230175    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:40.244360 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.244313    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 23:10:40.244516 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.244489    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:40.244647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.244528    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:40.244647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.244528    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 23:10:40.306898 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.306855    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 23:05:39 +0000 UTC" deadline="2028-04-14 16:01:25.38973543 +0000 UTC"
Sep 17 23:10:40.306898 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.306894    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13792h50m45.08284517s"
Sep 17 23:10:40.318112 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:40.318086    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-155.ec2.internal\" not found"
Sep 17 23:10:40.347210 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.347183    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:40.410002 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.409893    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:40.423573 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.423543    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:40.425233 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.425209    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal"
Sep 17 23:10:40.434376 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.434357    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 23:10:40.444827 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:40.444753    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52838b2473a80c0fab97ff455ee0daf5.slice/crio-7208e91d898466834c32f0be8fa4b58a861d9a9eeded5842b545dc91e826a291 WatchSource:0}: Error finding container 7208e91d898466834c32f0be8fa4b58a861d9a9eeded5842b545dc91e826a291: Status 404 returned error can't find the container with id 7208e91d898466834c32f0be8fa4b58a861d9a9eeded5842b545dc91e826a291
Sep 17 23:10:40.448131 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:40.448105    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbedd9a6fd6ace9132b55e90ebe8bf8b2.slice/crio-5bb86509914174ba8bbe586cac3ece9db2ef5c8a435a8d2616737efc44e682bb WatchSource:0}: Error finding container 5bb86509914174ba8bbe586cac3ece9db2ef5c8a435a8d2616737efc44e682bb: Status 404 returned error can't find the container with id 5bb86509914174ba8bbe586cac3ece9db2ef5c8a435a8d2616737efc44e682bb
Sep 17 23:10:40.455244 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:40.455229    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:10:41.283602 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.283553    2562 apiserver.go:52] "Watching apiserver"
Sep 17 23:10:41.292493 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.292453    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 23:10:41.294821 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.294787    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-d4lvj","openshift-dns/node-resolver-9ngsd","openshift-image-registry/node-ca-szjtn","openshift-multus/network-metrics-daemon-mpl4d","openshift-network-operator/iptables-alerter-94q8v","openshift-ovn-kubernetes/ovnkube-node-kwr95","kube-system/konnectivity-agent-s94pm","kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm","openshift-cluster-node-tuning-operator/tuned-9d6xx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal","openshift-multus/multus-additional-cni-plugins-d7lfv","openshift-multus/multus-z62jl","openshift-network-diagnostics/network-check-target-ttv8n"]
Sep 17 23:10:41.297578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.297545    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.300116 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.300092    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.300225 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.300193    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 23:10:41.300225 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.300202    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 23:10:41.300340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.300201    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-sj2vg\""
Sep 17 23:10:41.303029 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.303010    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 23:10:41.303129 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.303064    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 23:10:41.303192 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.303016    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.303438 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.303419    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.304644 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.304622    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.304739 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.304649    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f58g8\""
Sep 17 23:10:41.305584 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.305562    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:41.305679 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.305640    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:41.305778 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.305761    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.306121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.306102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n4jtd\""
Sep 17 23:10:41.306404 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.306380    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.306777 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.306729    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 23:10:41.309149 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.309127    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.309956 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.309932    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 23:10:41.311788 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.311770    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.312034 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.312011    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nxstj\""
Sep 17 23:10:41.312359 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.312342    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.312538 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.312520    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 23:10:41.313093 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.313033    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.314993 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.314958    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 23:10:41.315092 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.314992    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.315092 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.315072    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 23:10:41.316517 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.316369    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.316517 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.316505    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.316670 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.316604    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 23:10:41.316670 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.316651    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 23:10:41.318492 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.317070    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zc246\""
Sep 17 23:10:41.318492 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.317543    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.318492 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.317607    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:41.318492 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.318209    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 23:10:41.318492 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.318348    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.318933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.318502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.318933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.318748    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jsf4s\""
Sep 17 23:10:41.320511 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.320490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.322797 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.322775    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.323562 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.323544    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.323719 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.323700    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ktmvr\""
Sep 17 23:10:41.323806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.323753    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 23:10:41.324008 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.323992    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.325291 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db20b0b2-8082-4c9e-b2e9-f136c3b2efec-agent-certs\") pod \"konnectivity-agent-s94pm\" (UID: \"db20b0b2-8082-4c9e-b2e9-f136c3b2efec\") " pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.325386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-cni-binary-copy\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.325386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:41.325386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325364    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-kubelet\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.325572 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325389    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-etc-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.325572 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325429    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-env-overrides\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.325572 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021443d4-2c25-4789-9f74-acc54f1e1095-host\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.325572 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325477    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-systemd-units\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.325572 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-systemd\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.325572 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325552    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.325865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325578    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m6dz\" (UniqueName: \"kubernetes.io/projected/2d03d11b-533f-4d38-9ca5-c431f7e34f21-kube-api-access-4m6dz\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.325865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325605    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-cnibin\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.325987 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwlwj\" (UniqueName: \"kubernetes.io/projected/021443d4-2c25-4789-9f74-acc54f1e1095-kube-api-access-pwlwj\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.326224 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.325963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-run-netns\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.326224 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-log-socket\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.326224 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326177    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-cni-netd\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.326432 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326216    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovn-node-metrics-cert\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.326432 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326408    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-hostroot\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.326538 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326440    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz9dp\" (UniqueName: \"kubernetes.io/projected/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-kube-api-access-cz9dp\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.326538 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326472    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/021443d4-2c25-4789-9f74-acc54f1e1095-serviceca\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.326538 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h7tx\" (UniqueName: \"kubernetes.io/projected/94abe8fb-5a88-4039-884a-1397dec13323-kube-api-access-4h7tx\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:41.326696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.326541    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.330569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.330542    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 23:10:41.330659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.330582    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xhl8z\""
Sep 17 23:10:41.330879 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.330852    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:10:41.330966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.330913    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-ovn\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.330966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.330954    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:41.331178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.330962    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-cni-bin\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.331178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.330992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovnkube-config\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.331178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.331024    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-system-cni-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.331178 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.331063    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:41.332716 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-kubelet\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.332716 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9775c231-c04f-4955-bc8c-37a6b9777b47-host-slash\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.332716 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-cni-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.332954 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-var-lib-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.332954 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-node-log\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.332954 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332826    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db20b0b2-8082-4c9e-b2e9-f136c3b2efec-konnectivity-ca\") pod \"konnectivity-agent-s94pm\" (UID: \"db20b0b2-8082-4c9e-b2e9-f136c3b2efec\") " pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.332954 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-netns\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.332954 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.333243 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332963    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-cni-multus\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333243 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.332996    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-conf-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333243 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333047    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-daemon-config\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333243 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333101    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-etc-kubernetes\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333243 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333145    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9775c231-c04f-4955-bc8c-37a6b9777b47-iptables-alerter-script\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.333243 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjt6\" (UniqueName: \"kubernetes.io/projected/9775c231-c04f-4955-bc8c-37a6b9777b47-kube-api-access-vtjt6\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.333551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333250    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovnkube-script-lib\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.333551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333289    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-socket-dir-parent\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333318    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-k8s-cni-cncf-io\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333358    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-cni-bin\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-multus-certs\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.333551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-slash\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.333852 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.333793    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.337297 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.337271    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t9t5p\""
Sep 17 23:10:41.337499 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.337480    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 23:10:41.337648 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.337629    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 23:10:41.344208 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.344184    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bl2h8"
Sep 17 23:10:41.353320 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.353284    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bl2h8"
Sep 17 23:10:41.411376 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.411348    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 23:10:41.418578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.418519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal" event={"ID":"bedd9a6fd6ace9132b55e90ebe8bf8b2","Type":"ContainerStarted","Data":"5bb86509914174ba8bbe586cac3ece9db2ef5c8a435a8d2616737efc44e682bb"}
Sep 17 23:10:41.419890 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.419857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal" event={"ID":"52838b2473a80c0fab97ff455ee0daf5","Type":"ContainerStarted","Data":"7208e91d898466834c32f0be8fa4b58a861d9a9eeded5842b545dc91e826a291"}
Sep 17 23:10:41.434384 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434340    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovnkube-config\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.434384 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434386    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-system-cni-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434599 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-kubelet\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434599 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434445    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysctl-d\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.434599 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434500    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-lib-modules\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.434599 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-host\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.434599 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-netns\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysconfig\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434634    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-system-cni-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434641    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkzq\" (UniqueName: \"kubernetes.io/projected/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-kube-api-access-5vkzq\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-kubelet\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434694    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-netns\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ae1b512-2e62-465c-8d6f-8e782e948808-kubelet-config\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434735    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-conf-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434788    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-daemon-config\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434835    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-conf-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434843    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-etc-kubernetes\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434878    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-system-cni-dir\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434894    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-run-ovn-kubernetes\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434942    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-cnibin\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.434999    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-etc-kubernetes\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovnkube-script-lib\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435102    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db20b0b2-8082-4c9e-b2e9-f136c3b2efec-agent-certs\") pod \"konnectivity-agent-s94pm\" (UID: \"db20b0b2-8082-4c9e-b2e9-f136c3b2efec\") " pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435109    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovnkube-config\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435128    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-cni-binary-copy\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-var-lib-kubelet\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-tmp\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435226    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435284    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435309    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-cnibin\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435348    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwlwj\" (UniqueName: \"kubernetes.io/projected/021443d4-2c25-4789-9f74-acc54f1e1095-kube-api-access-pwlwj\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.435491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysctl-conf\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435396    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-sys\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435433    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-daemon-config\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-run-netns\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-cni-netd\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-cni-netd\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435531    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435580    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-cnibin\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-run-netns\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435635    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovnkube-script-lib\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovn-node-metrics-cert\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435687    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz9dp\" (UniqueName: \"kubernetes.io/projected/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-kube-api-access-cz9dp\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435755    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-tuned\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf9fz\" (UniqueName: \"kubernetes.io/projected/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-kube-api-access-mf9fz\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ae1b512-2e62-465c-8d6f-8e782e948808-dbus\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.436178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9775c231-c04f-4955-bc8c-37a6b9777b47-host-slash\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-run\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.435957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436005    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-socket-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-cni-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9775c231-c04f-4955-bc8c-37a6b9777b47-host-slash\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436070    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-var-lib-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436073    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-node-log\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436114    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-var-lib-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db20b0b2-8082-4c9e-b2e9-f136c3b2efec-konnectivity-ca\") pod \"konnectivity-agent-s94pm\" (UID: \"db20b0b2-8082-4c9e-b2e9-f136c3b2efec\") " pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436163    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-node-log\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-cni-multus\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-cni-dir\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.436832 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436240    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9775c231-c04f-4955-bc8c-37a6b9777b47-iptables-alerter-script\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-cni-multus\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436270    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjt6\" (UniqueName: \"kubernetes.io/projected/9775c231-c04f-4955-bc8c-37a6b9777b47-kube-api-access-vtjt6\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436081    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-cni-binary-copy\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p4w6\" (UniqueName: \"kubernetes.io/projected/3963701d-1a30-4685-99f4-d9a1bc8af35b-kube-api-access-9p4w6\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436506    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-kubernetes\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-socket-dir-parent\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-k8s-cni-cncf-io\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-k8s-cni-cncf-io\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436661    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-cni-bin\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-multus-certs\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-var-lib-cni-bin\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-host-run-multus-certs\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436776    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-registration-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/db20b0b2-8082-4c9e-b2e9-f136c3b2efec-konnectivity-ca\") pod \"konnectivity-agent-s94pm\" (UID: \"db20b0b2-8082-4c9e-b2e9-f136c3b2efec\") " pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436824    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-multus-socket-dir-parent\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436817    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9775c231-c04f-4955-bc8c-37a6b9777b47-iptables-alerter-script\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.437699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-slash\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436879    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436908    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-modprobe-d\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-kubelet\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.436976    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-etc-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-env-overrides\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021443d4-2c25-4789-9f74-acc54f1e1095-host\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56gj\" (UniqueName: \"kubernetes.io/projected/508404ff-8f01-4fff-8f1f-cffeb8563192-kube-api-access-d56gj\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-sys-fs\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.437086    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437088    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-etc-openvswitch\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-systemd-units\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-systemd\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437156    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-kubelet\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-systemd\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.437224    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:41.93714877 +0000 UTC m=+3.031675788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437240    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-systemd-units\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.438619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m6dz\" (UniqueName: \"kubernetes.io/projected/2d03d11b-533f-4d38-9ca5-c431f7e34f21-kube-api-access-4m6dz\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-device-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-log-socket\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437417    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-hostroot\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/021443d4-2c25-4789-9f74-acc54f1e1095-serviceca\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437444    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-slash\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437454    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/021443d4-2c25-4789-9f74-acc54f1e1095-host\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437468    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h7tx\" (UniqueName: \"kubernetes.io/projected/94abe8fb-5a88-4039-884a-1397dec13323-kube-api-access-4h7tx\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437503    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3963701d-1a30-4685-99f4-d9a1bc8af35b-hosts-file\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437515    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-log-socket\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437532    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3963701d-1a30-4685-99f4-d9a1bc8af35b-tmp-dir\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d03d11b-533f-4d38-9ca5-c431f7e34f21-env-overrides\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-systemd\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437628    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437657    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-ovn\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-cni-bin\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437687    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-hostroot\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437766    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-host-cni-bin\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.439478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.437813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d03d11b-533f-4d38-9ca5-c431f7e34f21-run-ovn\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.440442 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.438012    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/021443d4-2c25-4789-9f74-acc54f1e1095-serviceca\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.441205 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.441182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d03d11b-533f-4d38-9ca5-c431f7e34f21-ovn-node-metrics-cert\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.441333 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.441235    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/db20b0b2-8082-4c9e-b2e9-f136c3b2efec-agent-certs\") pod \"konnectivity-agent-s94pm\" (UID: \"db20b0b2-8082-4c9e-b2e9-f136c3b2efec\") " pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.447724 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.447690    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwlwj\" (UniqueName: \"kubernetes.io/projected/021443d4-2c25-4789-9f74-acc54f1e1095-kube-api-access-pwlwj\") pod \"node-ca-szjtn\" (UID: \"021443d4-2c25-4789-9f74-acc54f1e1095\") " pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.449412 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.449292    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz9dp\" (UniqueName: \"kubernetes.io/projected/fbf47e2b-0dfd-4a92-9042-8593f08a9a73-kube-api-access-cz9dp\") pod \"multus-z62jl\" (UID: \"fbf47e2b-0dfd-4a92-9042-8593f08a9a73\") " pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.449555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.449530    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjt6\" (UniqueName: \"kubernetes.io/projected/9775c231-c04f-4955-bc8c-37a6b9777b47-kube-api-access-vtjt6\") pod \"iptables-alerter-94q8v\" (UID: \"9775c231-c04f-4955-bc8c-37a6b9777b47\") " pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.452365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.452342    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m6dz\" (UniqueName: \"kubernetes.io/projected/2d03d11b-533f-4d38-9ca5-c431f7e34f21-kube-api-access-4m6dz\") pod \"ovnkube-node-kwr95\" (UID: \"2d03d11b-533f-4d38-9ca5-c431f7e34f21\") " pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.453081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.453040    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h7tx\" (UniqueName: \"kubernetes.io/projected/94abe8fb-5a88-4039-884a-1397dec13323-kube-api-access-4h7tx\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:41.538775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538689    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysctl-d\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.538775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538727    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-lib-modules\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.538775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-host\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.538775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538771    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysconfig\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkzq\" (UniqueName: \"kubernetes.io/projected/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-kube-api-access-5vkzq\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ae1b512-2e62-465c-8d6f-8e782e948808-kubelet-config\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-system-cni-dir\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538856    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-host\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-cnibin\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538902    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysctl-d\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538906    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-lib-modules\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538925    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-var-lib-kubelet\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538954    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9ae1b512-2e62-465c-8d6f-8e782e948808-kubelet-config\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-tmp\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538993    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysconfig\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-cnibin\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.538992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539034    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-system-cni-dir\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.539142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539058    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-var-lib-kubelet\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539071    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysctl-conf\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539093    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-sys\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539115    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539145    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539171    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-tuned\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf9fz\" (UniqueName: \"kubernetes.io/projected/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-kube-api-access-mf9fz\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539220    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ae1b512-2e62-465c-8d6f-8e782e948808-dbus\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539245    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-run\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-sysctl-conf\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539295    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539324    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-socket-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p4w6\" (UniqueName: \"kubernetes.io/projected/3963701d-1a30-4685-99f4-d9a1bc8af35b-kube-api-access-9p4w6\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-kubernetes\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539391    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9ae1b512-2e62-465c-8d6f-8e782e948808-dbus\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539413    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-registration-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.539148    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:41.539925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-sys\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-modprobe-d\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-run\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.539497    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret podName:9ae1b512-2e62-465c-8d6f-8e782e948808 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:42.039472789 +0000 UTC m=+3.133999819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret") pod "global-pull-secret-syncer-d4lvj" (UID: "9ae1b512-2e62-465c-8d6f-8e782e948808") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56gj\" (UniqueName: \"kubernetes.io/projected/508404ff-8f01-4fff-8f1f-cffeb8563192-kube-api-access-d56gj\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-sys-fs\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-device-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539743    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3963701d-1a30-4685-99f4-d9a1bc8af35b-hosts-file\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3963701d-1a30-4685-99f4-d9a1bc8af35b-tmp-dir\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539772    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539791    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-systemd\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539811    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-modprobe-d\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539818    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539830    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-sys-fs\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539934    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-systemd\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.540904 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.539970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540003    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-registration-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540048    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-socket-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540139    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3963701d-1a30-4685-99f4-d9a1bc8af35b-hosts-file\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540144    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3963701d-1a30-4685-99f4-d9a1bc8af35b-tmp-dir\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540197    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-kubernetes\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/508404ff-8f01-4fff-8f1f-cffeb8563192-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-device-dir\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540324    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/508404ff-8f01-4fff-8f1f-cffeb8563192-cni-binary-copy\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.541764 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.540331    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.542530 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.542506    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-etc-tuned\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.542643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.542625    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-tmp\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.552946 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.552885    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:41.552946 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.552929    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:41.552946 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.552948    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dlc5n for pod openshift-network-diagnostics/network-check-target-ttv8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:41.553294 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.553040    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n podName:c31f4265-c4c3-49a8-b13a-e84f31c0f6f2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:42.053016802 +0000 UTC m=+3.147543825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dlc5n" (UniqueName: "kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n") pod "network-check-target-ttv8n" (UID: "c31f4265-c4c3-49a8-b13a-e84f31c0f6f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:41.555763 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.555735    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf9fz\" (UniqueName: \"kubernetes.io/projected/3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e-kube-api-access-mf9fz\") pod \"tuned-9d6xx\" (UID: \"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e\") " pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.555929 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.555905    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkzq\" (UniqueName: \"kubernetes.io/projected/4bc5b66c-097f-42b2-8753-49ac2ecb72b1-kube-api-access-5vkzq\") pod \"aws-ebs-csi-driver-node-5mshm\" (UID: \"4bc5b66c-097f-42b2-8753-49ac2ecb72b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.556124 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.556106    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56gj\" (UniqueName: \"kubernetes.io/projected/508404ff-8f01-4fff-8f1f-cffeb8563192-kube-api-access-d56gj\") pod \"multus-additional-cni-plugins-d7lfv\" (UID: \"508404ff-8f01-4fff-8f1f-cffeb8563192\") " pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.556172 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.556134    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p4w6\" (UniqueName: \"kubernetes.io/projected/3963701d-1a30-4685-99f4-d9a1bc8af35b-kube-api-access-9p4w6\") pod \"node-resolver-9ngsd\" (UID: \"3963701d-1a30-4685-99f4-d9a1bc8af35b\") " pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.611973 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.611935    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:10:41.624022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.623998    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z62jl"
Sep 17 23:10:41.638951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.638916    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-szjtn"
Sep 17 23:10:41.645043 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.645021    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-94q8v"
Sep 17 23:10:41.650881 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.650862    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:10:41.660802 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.660780    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm"
Sep 17 23:10:41.668628 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.668607    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9ngsd"
Sep 17 23:10:41.675685 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.675667    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9d6xx"
Sep 17 23:10:41.682666 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.682645    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d7lfv"
Sep 17 23:10:41.843704 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.843624    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:41.943694 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:41.943647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:41.943890 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.943831    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:41.943984 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:41.943910    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:42.943889773 +0000 UTC m=+4.038416804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:42.044848 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:42.044813    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:42.045020 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.044965    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:42.045068 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.045041    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret podName:9ae1b512-2e62-465c-8d6f-8e782e948808 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.045022073 +0000 UTC m=+4.139549094 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret") pod "global-pull-secret-syncer-d4lvj" (UID: "9ae1b512-2e62-465c-8d6f-8e782e948808") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:42.145447 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:42.145355    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:42.145609 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.145504    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:42.145609 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.145524    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:42.145609 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.145534    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dlc5n for pod openshift-network-diagnostics/network-check-target-ttv8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:42.145609 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.145601    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n podName:c31f4265-c4c3-49a8-b13a-e84f31c0f6f2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:43.145583295 +0000 UTC m=+4.240110313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlc5n" (UniqueName: "kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n") pod "network-check-target-ttv8n" (UID: "c31f4265-c4c3-49a8-b13a-e84f31c0f6f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:42.358075 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:42.358026    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:41 +0000 UTC" deadline="2028-03-30 22:04:55.122293055 +0000 UTC"
Sep 17 23:10:42.358075 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:42.358064    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13438h54m12.764234427s"
Sep 17 23:10:42.493520 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.493484    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9775c231_c04f_4955_bc8c_37a6b9777b47.slice/crio-e54b1ccb5f4e96ab27ae783b2d3c1adc8020e0fdfe1d23fbc1c635a15660ce07 WatchSource:0}: Error finding container e54b1ccb5f4e96ab27ae783b2d3c1adc8020e0fdfe1d23fbc1c635a15660ce07: Status 404 returned error can't find the container with id e54b1ccb5f4e96ab27ae783b2d3c1adc8020e0fdfe1d23fbc1c635a15660ce07
Sep 17 23:10:42.495080 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.495058    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc5b66c_097f_42b2_8753_49ac2ecb72b1.slice/crio-febd366057be8427bc7455e20778ad97f17b4209a7c14281e088f10c72b085ef WatchSource:0}: Error finding container febd366057be8427bc7455e20778ad97f17b4209a7c14281e088f10c72b085ef: Status 404 returned error can't find the container with id febd366057be8427bc7455e20778ad97f17b4209a7c14281e088f10c72b085ef
Sep 17 23:10:42.497206 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.496764    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf47e2b_0dfd_4a92_9042_8593f08a9a73.slice/crio-f0871c3cdbe447088bf90dd1b9e047684de13eaa260fdeac1054628faf37351a WatchSource:0}: Error finding container f0871c3cdbe447088bf90dd1b9e047684de13eaa260fdeac1054628faf37351a: Status 404 returned error can't find the container with id f0871c3cdbe447088bf90dd1b9e047684de13eaa260fdeac1054628faf37351a
Sep 17 23:10:42.501300 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.501260    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3963701d_1a30_4685_99f4_d9a1bc8af35b.slice/crio-8fd028ae3e90ce25f17f54828bc8bf36966be30381d5bae55fae51fa2567c25f WatchSource:0}: Error finding container 8fd028ae3e90ce25f17f54828bc8bf36966be30381d5bae55fae51fa2567c25f: Status 404 returned error can't find the container with id 8fd028ae3e90ce25f17f54828bc8bf36966be30381d5bae55fae51fa2567c25f
Sep 17 23:10:42.503060 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.503034    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508404ff_8f01_4fff_8f1f_cffeb8563192.slice/crio-e7ce4943e91c9ca72da4d9180fe6642f72f042d0a9b50852740899f4121fda08 WatchSource:0}: Error finding container e7ce4943e91c9ca72da4d9180fe6642f72f042d0a9b50852740899f4121fda08: Status 404 returned error can't find the container with id e7ce4943e91c9ca72da4d9180fe6642f72f042d0a9b50852740899f4121fda08
Sep 17 23:10:42.503866 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.503842    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb20b0b2_8082_4c9e_b2e9_f136c3b2efec.slice/crio-0c0514aba8fee1c77cf6ab67203e4e7b3a2d9fab5bd4a1db38cc3001fa7d5ede WatchSource:0}: Error finding container 0c0514aba8fee1c77cf6ab67203e4e7b3a2d9fab5bd4a1db38cc3001fa7d5ede: Status 404 returned error can't find the container with id 0c0514aba8fee1c77cf6ab67203e4e7b3a2d9fab5bd4a1db38cc3001fa7d5ede
Sep 17 23:10:42.504811 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.504777    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021443d4_2c25_4789_9f74_acc54f1e1095.slice/crio-fe827149e2953d4b8246ab403633535f9b1de1665c69d997d085013709bcdf4c WatchSource:0}: Error finding container fe827149e2953d4b8246ab403633535f9b1de1665c69d997d085013709bcdf4c: Status 404 returned error can't find the container with id fe827149e2953d4b8246ab403633535f9b1de1665c69d997d085013709bcdf4c
Sep 17 23:10:42.509883 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:10:42.509669    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d03d11b_533f_4d38_9ca5_c431f7e34f21.slice/crio-203c75094c471b7168cf84da9eaf2beb103db4469d664c05de97bee7ccd871aa WatchSource:0}: Error finding container 203c75094c471b7168cf84da9eaf2beb103db4469d664c05de97bee7ccd871aa: Status 404 returned error can't find the container with id 203c75094c471b7168cf84da9eaf2beb103db4469d664c05de97bee7ccd871aa
Sep 17 23:10:42.593250 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:42.593078    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:42.895857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:42.895826    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 23:10:42.952840 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:42.952189    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:42.952840 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.952388    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:42.952840 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:42.952458    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:44.952438573 +0000 UTC m=+6.046965610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:43.053362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.052596    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:43.053362 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.052846    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:43.053362 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.052917    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret podName:9ae1b512-2e62-465c-8d6f-8e782e948808 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:45.052897708 +0000 UTC m=+6.147424734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret") pod "global-pull-secret-syncer-d4lvj" (UID: "9ae1b512-2e62-465c-8d6f-8e782e948808") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:43.153843 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.153758    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:43.154077 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.153999    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:43.154077 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.154021    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:43.154077 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.154035    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dlc5n for pod openshift-network-diagnostics/network-check-target-ttv8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:43.154250 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.154099    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n podName:c31f4265-c4c3-49a8-b13a-e84f31c0f6f2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:45.154078849 +0000 UTC m=+6.248605871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlc5n" (UniqueName: "kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n") pod "network-check-target-ttv8n" (UID: "c31f4265-c4c3-49a8-b13a-e84f31c0f6f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:43.358596 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.358436    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 23:05:41 +0000 UTC" deadline="2028-04-30 12:40:51.326567069 +0000 UTC"
Sep 17 23:10:43.358596 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.358506    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14173h30m7.968065762s"
Sep 17 23:10:43.420905 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.420190    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:43.420905 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.420325    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:43.420905 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.420766    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:43.420905 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.420857    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:43.421590 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.421348    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:43.421590 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:43.421501    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:43.443863 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.443783    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerStarted","Data":"e7ce4943e91c9ca72da4d9180fe6642f72f042d0a9b50852740899f4121fda08"}
Sep 17 23:10:43.451530 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.451448    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s94pm" event={"ID":"db20b0b2-8082-4c9e-b2e9-f136c3b2efec","Type":"ContainerStarted","Data":"0c0514aba8fee1c77cf6ab67203e4e7b3a2d9fab5bd4a1db38cc3001fa7d5ede"}
Sep 17 23:10:43.454193 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.454098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm" event={"ID":"4bc5b66c-097f-42b2-8753-49ac2ecb72b1","Type":"ContainerStarted","Data":"febd366057be8427bc7455e20778ad97f17b4209a7c14281e088f10c72b085ef"}
Sep 17 23:10:43.461498 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.461470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-94q8v" event={"ID":"9775c231-c04f-4955-bc8c-37a6b9777b47","Type":"ContainerStarted","Data":"e54b1ccb5f4e96ab27ae783b2d3c1adc8020e0fdfe1d23fbc1c635a15660ce07"}
Sep 17 23:10:43.466835 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.466806    2562 generic.go:356] "Generic (PLEG): container finished" podID="bedd9a6fd6ace9132b55e90ebe8bf8b2" containerID="0423225222ab23e55c3d853143da6526b6e3fe71f268e04fca4c697a89afe592" exitCode=0
Sep 17 23:10:43.466933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.466894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal" event={"ID":"bedd9a6fd6ace9132b55e90ebe8bf8b2","Type":"ContainerDied","Data":"0423225222ab23e55c3d853143da6526b6e3fe71f268e04fca4c697a89afe592"}
Sep 17 23:10:43.486423 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.484946    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"203c75094c471b7168cf84da9eaf2beb103db4469d664c05de97bee7ccd871aa"}
Sep 17 23:10:43.535851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.535794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9d6xx" event={"ID":"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e","Type":"ContainerStarted","Data":"ac7b7bbf00df70e6cf591b6252aa429cc982c7922a6bebe4c479f1cddcd1b3de"}
Sep 17 23:10:43.549618 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.549514    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szjtn" event={"ID":"021443d4-2c25-4789-9f74-acc54f1e1095","Type":"ContainerStarted","Data":"fe827149e2953d4b8246ab403633535f9b1de1665c69d997d085013709bcdf4c"}
Sep 17 23:10:43.555901 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.555873    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9ngsd" event={"ID":"3963701d-1a30-4685-99f4-d9a1bc8af35b","Type":"ContainerStarted","Data":"8fd028ae3e90ce25f17f54828bc8bf36966be30381d5bae55fae51fa2567c25f"}
Sep 17 23:10:43.573660 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.573608    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z62jl" event={"ID":"fbf47e2b-0dfd-4a92-9042-8593f08a9a73","Type":"ContainerStarted","Data":"f0871c3cdbe447088bf90dd1b9e047684de13eaa260fdeac1054628faf37351a"}
Sep 17 23:10:43.582434 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:43.581603    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal" event={"ID":"52838b2473a80c0fab97ff455ee0daf5","Type":"ContainerStarted","Data":"13ee1966000bd9ada97a625438a3daa8c499d43e2cc4bbc1b3aed5576a36fce3"}
Sep 17 23:10:44.605472 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:44.605435    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal" event={"ID":"bedd9a6fd6ace9132b55e90ebe8bf8b2","Type":"ContainerStarted","Data":"88c75f41d5456f04c12b0bd26a229f23a9ea6af2ffd73b9a87b34ea9264715d3"}
Sep 17 23:10:44.621892 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:44.621827    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-155.ec2.internal" podStartSLOduration=3.62180349 podStartE2EDuration="3.62180349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:43.598141926 +0000 UTC m=+4.692668967" watchObservedRunningTime="2026-09-17 23:10:44.62180349 +0000 UTC m=+5.716330531"
Sep 17 23:10:44.622179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:44.622144    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-155.ec2.internal" podStartSLOduration=3.622135453 podStartE2EDuration="3.622135453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:10:44.62122133 +0000 UTC m=+5.715748370" watchObservedRunningTime="2026-09-17 23:10:44.622135453 +0000 UTC m=+5.716662491"
Sep 17 23:10:44.975001 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:44.974519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:44.975001 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:44.974707    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:44.975001 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:44.974773    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:48.974753026 +0000 UTC m=+10.069280049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:45.075212 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:45.075175    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:45.075410 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.075318    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:45.075410 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.075389    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret podName:9ae1b512-2e62-465c-8d6f-8e782e948808 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.075369867 +0000 UTC m=+10.169896888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret") pod "global-pull-secret-syncer-d4lvj" (UID: "9ae1b512-2e62-465c-8d6f-8e782e948808") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:45.176418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:45.176379    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:45.176625 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.176605    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:45.176712 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.176633    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:45.176712 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.176648    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dlc5n for pod openshift-network-diagnostics/network-check-target-ttv8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:45.176828 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.176714    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n podName:c31f4265-c4c3-49a8-b13a-e84f31c0f6f2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:49.176693031 +0000 UTC m=+10.271220072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlc5n" (UniqueName: "kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n") pod "network-check-target-ttv8n" (UID: "c31f4265-c4c3-49a8-b13a-e84f31c0f6f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:45.415051 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:45.414048    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:45.415051 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.414196    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:45.415051 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:45.414620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:45.415051 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.414734    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:45.415051 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:45.414823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:45.415051 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:45.414912    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:47.416731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:47.415789    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:47.416731 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:47.415932    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:47.416731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:47.416427    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:47.416731 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:47.416539    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:47.416731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:47.416597    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:47.416731 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:47.416670    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:49.011960 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:49.011311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:49.011960 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.011463    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.011960 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.011529    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:57.011509018 +0000 UTC m=+18.106036047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:49.112098 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:49.112057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:49.112275 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.112239    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.112337 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.112326    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret podName:9ae1b512-2e62-465c-8d6f-8e782e948808 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:57.112304499 +0000 UTC m=+18.206831515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret") pod "global-pull-secret-syncer-d4lvj" (UID: "9ae1b512-2e62-465c-8d6f-8e782e948808") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:49.212816 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:49.212763    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:49.213006 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.212953    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:49.213006 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.212970    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:49.213006 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.212979    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dlc5n for pod openshift-network-diagnostics/network-check-target-ttv8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.213145 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.213034    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n podName:c31f4265-c4c3-49a8-b13a-e84f31c0f6f2 nodeName:}" failed. No retries permitted until 2026-09-17 23:10:57.21301634 +0000 UTC m=+18.307543357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlc5n" (UniqueName: "kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n") pod "network-check-target-ttv8n" (UID: "c31f4265-c4c3-49a8-b13a-e84f31c0f6f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:49.415109 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:49.414596    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:49.415109 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.414709    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:49.415109 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:49.414725    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:49.415109 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.414840    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:49.415109 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:49.414888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:49.415109 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:49.414993    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:51.413490 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:51.413340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:51.413490 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:51.413377    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:51.413490 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:51.413340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:51.413490 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:51.413475    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:51.414098 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:51.413535    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:51.414469 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:51.414428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:53.413141 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:53.413102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:53.413141 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:53.413146    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:53.413686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:53.413102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:53.413686 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:53.413229    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:53.413686 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:53.413303    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:53.413686 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:53.413348    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:55.413624 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:55.413588    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:55.413624 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:55.413625    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:55.414158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:55.413588    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:55.414158 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:55.413710    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:55.414158 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:55.413786    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:55.414158 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:55.413891    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:57.068235 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:57.068196    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:57.068679 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.068396    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:57.068679 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.068475    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.068454264 +0000 UTC m=+34.162981321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:10:57.169541 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:57.169505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:57.169708 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.169639    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:57.169708 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.169698    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret podName:9ae1b512-2e62-465c-8d6f-8e782e948808 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.16968441 +0000 UTC m=+34.264211424 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret") pod "global-pull-secret-syncer-d4lvj" (UID: "9ae1b512-2e62-465c-8d6f-8e782e948808") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:10:57.270811 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:57.270766    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:57.271003 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.270982    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:10:57.271055 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.271009    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:10:57.271055 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.271019    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dlc5n for pod openshift-network-diagnostics/network-check-target-ttv8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:57.271123 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.271068    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n podName:c31f4265-c4c3-49a8-b13a-e84f31c0f6f2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:13.271055508 +0000 UTC m=+34.365582523 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlc5n" (UniqueName: "kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n") pod "network-check-target-ttv8n" (UID: "c31f4265-c4c3-49a8-b13a-e84f31c0f6f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:10:57.412797 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:57.412719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:57.412957 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:57.412719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:57.412957 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.412827    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:57.412957 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:57.412719    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:57.412957 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.412893    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:57.413130 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:57.413010    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:10:59.413849 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:59.413817    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:10:59.414272 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:59.413939    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:10:59.414272 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:59.414022    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:10:59.414272 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:59.414128    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:10:59.414272 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:10:59.414176    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:10:59.414272 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:10:59.414249    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:01.413849 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.413439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:01.414488 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.413439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:01.414488 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:01.413984    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:01.414488 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.413502    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:01.414488 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:01.414054    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:01.414488 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:01.414161    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:01.644372 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.644279    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z62jl" event={"ID":"fbf47e2b-0dfd-4a92-9042-8593f08a9a73","Type":"ContainerStarted","Data":"5b7b92b17ecea6eebee8e2144205046682be54799515f18a542446cc9aaaab99"}
Sep 17 23:11:01.645658 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.645634    2562 generic.go:356] "Generic (PLEG): container finished" podID="508404ff-8f01-4fff-8f1f-cffeb8563192" containerID="08f3ce72a8ecfed0a72b0c9a26a41682837060466d690e3643265fefd528ee6a" exitCode=0
Sep 17 23:11:01.645787 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.645700    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerDied","Data":"08f3ce72a8ecfed0a72b0c9a26a41682837060466d690e3643265fefd528ee6a"}
Sep 17 23:11:01.647295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.647252    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s94pm" event={"ID":"db20b0b2-8082-4c9e-b2e9-f136c3b2efec","Type":"ContainerStarted","Data":"0bd865cc49e84875041fc50329f4f4caa5803011d9513b88a9b037e94e4254cf"}
Sep 17 23:11:01.648787 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.648765    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm" event={"ID":"4bc5b66c-097f-42b2-8753-49ac2ecb72b1","Type":"ContainerStarted","Data":"e5aa989c3db0b1ccc472f3f21d3cbafa551b01d7aa65f021975ae0726a3a1812"}
Sep 17 23:11:01.651874 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.651852    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"430d93d9b231cf54830c7c87d9e4f05d25fbff2de7239765b272eae595a10a40"}
Sep 17 23:11:01.651989 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.651942    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"dcce04973b76d4a11e0582938a5bde4a5316fdb61c66cc5e263fb7058f37e174"}
Sep 17 23:11:01.652052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.651997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"0a43e1231d45c8102f7c7443bc273fbfcc5fdebc7f8d47777eae9c2af1d38881"}
Sep 17 23:11:01.652052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.652045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"cc3280fc7674914eb0cead84d765caa91c27b7e2bba10702ccbdf9f566ef9aa1"}
Sep 17 23:11:01.652154 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.652094    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"ca360e16b49f0ee32bd370029201862fa9f67459b7dfa2d972711e1b9566d4b2"}
Sep 17 23:11:01.652154 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.652143    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"d949bbad8b1575142c5620a3f4f8a18e415393c6c270782228f2f07a2ce21cf4"}
Sep 17 23:11:01.653682 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.653602    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9d6xx" event={"ID":"3a465b3b-848b-45ae-ba5e-6a5fcdfd6c6e","Type":"ContainerStarted","Data":"71531a5dd53766991c22500e4bbe88c9439c13cc788f3472172ae3f945cffa4e"}
Sep 17 23:11:01.656270 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.656083    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-szjtn" event={"ID":"021443d4-2c25-4789-9f74-acc54f1e1095","Type":"ContainerStarted","Data":"4f4b21bd7c36e39499a5b45bd0064a91b869068783ce756cdeeb012ebeb0e527"}
Sep 17 23:11:01.658060 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.658040    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9ngsd" event={"ID":"3963701d-1a30-4685-99f4-d9a1bc8af35b","Type":"ContainerStarted","Data":"9b5f1fa84c2d6083994d184f5f991d561fa33934bec518b7308ad6214d34459b"}
Sep 17 23:11:01.661508 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.661462    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z62jl" podStartSLOduration=3.371841104 podStartE2EDuration="21.661444514s" totalImagesPullingTime="18.28960341s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.660648668 +0000 UTC m=+22.755175703" watchObservedRunningTime="2026-09-17 23:11:01.661444514 +0000 UTC m=+22.755971550"
Sep 17 23:11:01.694881 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.694829    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s94pm" podStartSLOduration=10.647399172 podStartE2EDuration="21.694810534s" totalImagesPullingTime="11.047411362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.694348156 +0000 UTC m=+22.788875191" watchObservedRunningTime="2026-09-17 23:11:01.694810534 +0000 UTC m=+22.789337570"
Sep 17 23:11:01.724009 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.723956    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9ngsd" podStartSLOduration=3.509481676 podStartE2EDuration="21.723936548s" totalImagesPullingTime="18.214454872s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.723681154 +0000 UTC m=+22.818208189" watchObservedRunningTime="2026-09-17 23:11:01.723936548 +0000 UTC m=+22.818463583"
Sep 17 23:11:01.724191 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.724038    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-szjtn" podStartSLOduration=3.515272226 podStartE2EDuration="21.724033215s" totalImagesPullingTime="18.208760989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.708768273 +0000 UTC m=+22.803295310" watchObservedRunningTime="2026-09-17 23:11:01.724033215 +0000 UTC m=+22.818560251"
Sep 17 23:11:01.743994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.743945    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9d6xx" podStartSLOduration=3.533415108 podStartE2EDuration="21.743923928s" totalImagesPullingTime="18.21050882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:01.743629344 +0000 UTC m=+22.838156381" watchObservedRunningTime="2026-09-17 23:11:01.743923928 +0000 UTC m=+22.838450964"
Sep 17 23:11:01.899926 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.899838    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:11:01.900407 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:01.900391    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:11:02.303491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.303468    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 23:11:02.488344 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.488192    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T23:11:02.303490725Z","UUID":"9805b9a3-d892-417a-9e6b-f34056946201","Handler":null,"Name":"","Endpoint":""}
Sep 17 23:11:02.489985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.489960    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 23:11:02.490113 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.489995    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 23:11:02.662534 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.662499    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm" event={"ID":"4bc5b66c-097f-42b2-8753-49ac2ecb72b1","Type":"ContainerStarted","Data":"cac421d29f0ddc213293b42a467dea8144bef806702ea5dd0a2e0d580f6ee0e4"}
Sep 17 23:11:02.664305 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.664197    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-94q8v" event={"ID":"9775c231-c04f-4955-bc8c-37a6b9777b47","Type":"ContainerStarted","Data":"044810220e78bc36513960ba56fdd72af36463587c98f38e7932c502a0a031b8"}
Sep 17 23:11:02.664754 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.664686    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:11:02.665157 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.665138    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s94pm"
Sep 17 23:11:02.693925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:02.693860    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-94q8v" podStartSLOduration=4.470943825 podStartE2EDuration="22.693841778s" totalImagesPullingTime="18.222897953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:02.693266648 +0000 UTC m=+23.787793684" watchObservedRunningTime="2026-09-17 23:11:02.693841778 +0000 UTC m=+23.788368814"
Sep 17 23:11:03.413571 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:03.413324    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:03.413707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:03.413323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:03.413707 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:03.413685    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:03.413814 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:03.413459    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:03.413814 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:03.413767    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:03.413935 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:03.413844    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:03.669779 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:03.669680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm" event={"ID":"4bc5b66c-097f-42b2-8753-49ac2ecb72b1","Type":"ContainerStarted","Data":"5316564c72c2e1c608c896bc03d7b6a5afc6eab2abec404d8a790d94081acc7b"}
Sep 17 23:11:03.673706 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:03.673591    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"cc856539eddfae38d4b682f24baadc0419ca5bd744c4e76801b5976b49f14524"}
Sep 17 23:11:03.686650 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:03.686593    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mshm" podStartSLOduration=3.283787435 podStartE2EDuration="23.68657716s" totalImagesPullingTime="20.402789725s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:03.686013596 +0000 UTC m=+24.780540631" watchObservedRunningTime="2026-09-17 23:11:03.68657716 +0000 UTC m=+24.781104196"
Sep 17 23:11:05.413270 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:05.413232    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:05.413806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:05.413232    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:05.413806 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:05.413374    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:05.413806 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:05.413451    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:05.413806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:05.413238    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:05.413806 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:05.413532    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:06.681462 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.681263    2562 generic.go:356] "Generic (PLEG): container finished" podID="508404ff-8f01-4fff-8f1f-cffeb8563192" containerID="37e9c52e851852e579dc2cd8352993f833290a62ec75ebbaa3929a8cb58760cb" exitCode=0
Sep 17 23:11:06.681851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.681343    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerDied","Data":"37e9c52e851852e579dc2cd8352993f833290a62ec75ebbaa3929a8cb58760cb"}
Sep 17 23:11:06.685494 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.685469    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" event={"ID":"2d03d11b-533f-4d38-9ca5-c431f7e34f21","Type":"ContainerStarted","Data":"67ce7ea89b2d23ee80138aca363ad54408b15da05960c74508ff87ff0119d6a6"}
Sep 17 23:11:06.685992 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.685801    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:11:06.685992 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.685875    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:11:06.685992 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.685938    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:11:06.700796 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.700776    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:11:06.700876 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.700861    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:11:06.726290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:06.726254    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95" podStartSLOduration=8.460693338 podStartE2EDuration="26.726243161s" totalImagesPullingTime="18.265549823s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:06.724337186 +0000 UTC m=+27.818864222" watchObservedRunningTime="2026-09-17 23:11:06.726243161 +0000 UTC m=+27.820770196"
Sep 17 23:11:07.413242 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.413211    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:07.413456 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.413261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:07.413456 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.413376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:07.413456 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:07.413387    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:07.413616 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:07.413454    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:07.413616 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:07.413522    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:07.643480 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.643447    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d4lvj"]
Sep 17 23:11:07.646537 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.646511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttv8n"]
Sep 17 23:11:07.647399 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.647383    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpl4d"]
Sep 17 23:11:07.690081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.689991    2562 generic.go:356] "Generic (PLEG): container finished" podID="508404ff-8f01-4fff-8f1f-cffeb8563192" containerID="2f2a5212f291e28c1f66f710a290416c635fb3d152191ee3f0af5e51ebd49d28" exitCode=0
Sep 17 23:11:07.690752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.690098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:07.690752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.690097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerDied","Data":"2f2a5212f291e28c1f66f710a290416c635fb3d152191ee3f0af5e51ebd49d28"}
Sep 17 23:11:07.690752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.690152    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:07.690752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:07.690240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:07.690752 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:07.690354    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:07.690752 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:07.690404    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:07.691227 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:07.690829    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:08.695022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:08.694797    2562 generic.go:356] "Generic (PLEG): container finished" podID="508404ff-8f01-4fff-8f1f-cffeb8563192" containerID="c66dc7e0fdb651ff3dbacec3bb9b6fb9bd91a55f8440978c12c3231c99b39afa" exitCode=0
Sep 17 23:11:08.695513 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:08.694866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerDied","Data":"c66dc7e0fdb651ff3dbacec3bb9b6fb9bd91a55f8440978c12c3231c99b39afa"}
Sep 17 23:11:09.414288 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:09.414255    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:09.414455 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:09.414335    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:09.414455 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:09.414366    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:09.414455 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:09.414382    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:09.414455 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:09.414440    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:09.414653 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:09.414527    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:11.413640 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:11.413608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:11.414095 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:11.413663    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:11.414095 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:11.413675    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:11.414095 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:11.413783    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:11.414095 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:11.413879    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:11.414095 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:11.414010    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:13.087093 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.086912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:13.087093 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.087013    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:13.087093 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.087079    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.08705734 +0000 UTC m=+66.181584353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 23:11:13.187882 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.187848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:13.188027 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.187961    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:13.188027 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.188009    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret podName:9ae1b512-2e62-465c-8d6f-8e782e948808 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.187994625 +0000 UTC m=+66.282521638 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret") pod "global-pull-secret-syncer-d4lvj" (UID: "9ae1b512-2e62-465c-8d6f-8e782e948808") : object "kube-system"/"original-pull-secret" not registered
Sep 17 23:11:13.289793 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.289768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:13.289931 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.289856    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 23:11:13.289931 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.289874    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 23:11:13.289931 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.289885    2562 projected.go:196] Error preparing data for projected volume kube-api-access-dlc5n for pod openshift-network-diagnostics/network-check-target-ttv8n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:13.290068 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.289955    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n podName:c31f4265-c4c3-49a8-b13a-e84f31c0f6f2 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.289942208 +0000 UTC m=+66.384469222 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dlc5n" (UniqueName: "kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n") pod "network-check-target-ttv8n" (UID: "c31f4265-c4c3-49a8-b13a-e84f31c0f6f2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 23:11:13.413029 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.412993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:13.413203 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.412993    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:13.413203 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.413126    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:13.413203 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.413121    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ttv8n" podUID="c31f4265-c4c3-49a8-b13a-e84f31c0f6f2"
Sep 17 23:11:13.413366 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.413216    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4lvj" podUID="9ae1b512-2e62-465c-8d6f-8e782e948808"
Sep 17 23:11:13.413366 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.413284    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:11:13.693693 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.693655    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-155.ec2.internal" event="NodeReady"
Sep 17 23:11:13.693849 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.693768    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 23:11:13.709552 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.709521    2562 generic.go:356] "Generic (PLEG): container finished" podID="508404ff-8f01-4fff-8f1f-cffeb8563192" containerID="192dae2cb6e57e24bd8db2388d864c9608870fd68750b16e77b952d1f2b181d2" exitCode=0
Sep 17 23:11:13.709703 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.709558    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerDied","Data":"192dae2cb6e57e24bd8db2388d864c9608870fd68750b16e77b952d1f2b181d2"}
Sep 17 23:11:13.730643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.730611    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5874477fdf-6xtqp"]
Sep 17 23:11:13.755429 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.755400    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nc7w8"]
Sep 17 23:11:13.755595 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.755526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.758576 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.758548    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9dbhz\""
Sep 17 23:11:13.758576 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.758555    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 23:11:13.758875 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.758850    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 23:11:13.758986 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.758964    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 23:11:13.765470 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.765200    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 23:11:13.767355 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.767330    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5874477fdf-6xtqp"]
Sep 17 23:11:13.767467 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.767375    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-m8nq7"]
Sep 17 23:11:13.767467 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.767439    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.770728 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.770697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 23:11:13.770933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.770893    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvr9f\""
Sep 17 23:11:13.770992 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.770964    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 23:11:13.783632 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.783605    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nc7w8"]
Sep 17 23:11:13.783763 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.783650    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8nq7"]
Sep 17 23:11:13.783763 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.783742    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:13.786315 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.786288    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 23:11:13.786442 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.786352    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 23:11:13.786442 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.786376    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 23:11:13.786442 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.786387    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fsw47\""
Sep 17 23:11:13.894110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngl4\" (UniqueName: \"kubernetes.io/projected/613efdc2-51ea-45ce-b410-861640d21bbf-kube-api-access-gngl4\") pod \"ingress-canary-m8nq7\" (UID: \"613efdc2-51ea-45ce-b410-861640d21bbf\") " pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:13.894273 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894115    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d957c71e-eb0d-4976-97d7-21058a4171c8-ca-trust-extracted\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894273 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/613efdc2-51ea-45ce-b410-861640d21bbf-cert\") pod \"ingress-canary-m8nq7\" (UID: \"613efdc2-51ea-45ce-b410-861640d21bbf\") " pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:13.894273 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894171    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-installation-pull-secrets\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894273 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894210    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-config-volume\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.894273 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-certificates\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894273 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8jgw\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-kube-api-access-z8jgw\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894280    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-trusted-ca\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894295    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-image-registry-private-configuration\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-bound-sa-token\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894333    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.894555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894387    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xknnw\" (UniqueName: \"kubernetes.io/projected/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-kube-api-access-xknnw\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.894555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894509    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.894555 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.894545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-tmp-dir\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.995765 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.995719    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d957c71e-eb0d-4976-97d7-21058a4171c8-ca-trust-extracted\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.995985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.995781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/613efdc2-51ea-45ce-b410-861640d21bbf-cert\") pod \"ingress-canary-m8nq7\" (UID: \"613efdc2-51ea-45ce-b410-861640d21bbf\") " pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:13.995985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.995809    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-installation-pull-secrets\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.995985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.995835    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-config-volume\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.995985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.995868    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-certificates\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.995985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.995891    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8jgw\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-kube-api-access-z8jgw\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.996297 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996269    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d957c71e-eb0d-4976-97d7-21058a4171c8-ca-trust-extracted\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.996381 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-trusted-ca\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.996441 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-image-registry-private-configuration\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.996499 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996435    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-bound-sa-token\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.996499 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.996590 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996524    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-certificates\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.996590 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.996554    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:13.996590 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.996569    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:11:13.996728 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-config-volume\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.996728 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.996623    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:14.496605528 +0000 UTC m=+35.591132563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:11:13.996728 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xknnw\" (UniqueName: \"kubernetes.io/projected/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-kube-api-access-xknnw\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.996873 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996786    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.996873 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996810    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-tmp-dir\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:13.996873 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.996840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngl4\" (UniqueName: \"kubernetes.io/projected/613efdc2-51ea-45ce-b410-861640d21bbf-kube-api-access-gngl4\") pod \"ingress-canary-m8nq7\" (UID: \"613efdc2-51ea-45ce-b410-861640d21bbf\") " pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:13.997053 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.996930    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:13.997053 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:13.996986    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:14.496966887 +0000 UTC m=+35.591493917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:13.997166 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.997128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-trusted-ca\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:13.997218 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:13.997211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-tmp-dir\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:14.000230 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.000206    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/613efdc2-51ea-45ce-b410-861640d21bbf-cert\") pod \"ingress-canary-m8nq7\" (UID: \"613efdc2-51ea-45ce-b410-861640d21bbf\") " pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:14.000230 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.000223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-installation-pull-secrets\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:14.000364 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.000223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-image-registry-private-configuration\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:14.007017 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.006988    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xknnw\" (UniqueName: \"kubernetes.io/projected/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-kube-api-access-xknnw\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:14.007229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.007209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8jgw\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-kube-api-access-z8jgw\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:14.007616 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.007601    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-bound-sa-token\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:14.007857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.007842    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngl4\" (UniqueName: \"kubernetes.io/projected/613efdc2-51ea-45ce-b410-861640d21bbf-kube-api-access-gngl4\") pod \"ingress-canary-m8nq7\" (UID: \"613efdc2-51ea-45ce-b410-861640d21bbf\") " pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:14.094142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.093954    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-m8nq7"
Sep 17 23:11:14.450990 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.450804    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-m8nq7"]
Sep 17 23:11:14.454118 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:11:14.454085    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613efdc2_51ea_45ce_b410_861640d21bbf.slice/crio-9879dac50e7feed090425cbf60febcc162fe968b47dfbb27dd43f3db75498bee WatchSource:0}: Error finding container 9879dac50e7feed090425cbf60febcc162fe968b47dfbb27dd43f3db75498bee: Status 404 returned error can't find the container with id 9879dac50e7feed090425cbf60febcc162fe968b47dfbb27dd43f3db75498bee
Sep 17 23:11:14.500296 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.500267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:14.500463 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.500311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:14.500463 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:14.500407    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:14.500463 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:14.500413    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:14.500463 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:14.500431    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:11:14.500463 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:14.500463    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:15.500450332 +0000 UTC m=+36.594977346 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:14.500644 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:14.500475    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:15.500469592 +0000 UTC m=+36.594996605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:11:14.714750 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.714673    2562 generic.go:356] "Generic (PLEG): container finished" podID="508404ff-8f01-4fff-8f1f-cffeb8563192" containerID="4f7349de9cc8f7820761f24224d2fe825e28659d759d3fd7871c91cc789c0c21" exitCode=0
Sep 17 23:11:14.714890 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.714751    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerDied","Data":"4f7349de9cc8f7820761f24224d2fe825e28659d759d3fd7871c91cc789c0c21"}
Sep 17 23:11:14.715928 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:14.715884    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8nq7" event={"ID":"613efdc2-51ea-45ce-b410-861640d21bbf","Type":"ContainerStarted","Data":"9879dac50e7feed090425cbf60febcc162fe968b47dfbb27dd43f3db75498bee"}
Sep 17 23:11:15.412958 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.412920    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:15.413850 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.412920    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:15.413850 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.412920    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:15.416046 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.416019    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:15.416222 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.416019    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qw6rm\""
Sep 17 23:11:15.417645 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.417620    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:15.417785 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.417672    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:15.417785 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.417697    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mx9lw\""
Sep 17 23:11:15.417785 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.417728    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:15.508399 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.508365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:15.508587 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.508426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:15.508587 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:15.508571    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:15.508692 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:15.508594    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:11:15.508692 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:15.508634    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:15.508692 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:15.508666    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:17.508641737 +0000 UTC m=+38.603168767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:11:15.508840 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:15.508699    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:17.508681134 +0000 UTC m=+38.603208149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:15.722964 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.722864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" event={"ID":"508404ff-8f01-4fff-8f1f-cffeb8563192","Type":"ContainerStarted","Data":"03c3af28b8a1ea22476bcb2d14687ec93c67656b9e31cb1e9f7a7bca086055e1"}
Sep 17 23:11:15.745439 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:15.745386    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d7lfv" podStartSLOduration=7.68013585 podStartE2EDuration="35.74536811s" totalImagesPullingTime="28.06523226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:15.744842561 +0000 UTC m=+36.839369597" watchObservedRunningTime="2026-09-17 23:11:15.74536811 +0000 UTC m=+36.839895145"
Sep 17 23:11:17.526378 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:17.526109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:17.526378 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:17.526331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:17.526378 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:17.526271    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:17.526856 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:17.526392    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:11:17.526856 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:17.526429    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:17.526856 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:17.526456    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:21.526438219 +0000 UTC m=+42.620965252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:11:17.526856 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:17.526470    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:21.526463602 +0000 UTC m=+42.620990615 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:17.729980 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:17.729943    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-m8nq7" event={"ID":"613efdc2-51ea-45ce-b410-861640d21bbf","Type":"ContainerStarted","Data":"a11bba2697e25485b1609a3a2e2746ec705b228abbde71c4c05d5e60c53b48c7"}
Sep 17 23:11:17.745211 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:17.745163    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-m8nq7" podStartSLOduration=2.452947425 podStartE2EDuration="4.745147422s" totalImagesPullingTime="2.292199997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:17.744412105 +0000 UTC m=+38.838939140" watchObservedRunningTime="2026-09-17 23:11:17.745147422 +0000 UTC m=+38.839674448"
Sep 17 23:11:21.557212 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:21.557167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:21.557565 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:21.557228    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:21.557565 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:21.557330    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:21.557565 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:21.557342    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:21.557565 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:21.557349    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:11:21.557565 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:21.557398    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:29.557383092 +0000 UTC m=+50.651910106 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:21.557565 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:21.557411    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:29.557405179 +0000 UTC m=+50.651932192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:11:29.615032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:29.614990    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:29.615507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:29.615058    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:29.615507 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:29.615151    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:29.615507 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:29.615166    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:29.615507 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:29.615171    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:11:29.615507 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:29.615232    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.615214014 +0000 UTC m=+66.709741027 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:29.615507 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:29.615325    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:11:45.615311209 +0000 UTC m=+66.709838225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:11:38.705774 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:38.705746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kwr95"
Sep 17 23:11:45.122622 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.122581    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:11:45.125412 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.125391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 23:11:45.133707 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:45.133681    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:11:45.133780 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:45.133749    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:49.133732793 +0000 UTC m=+130.228259812 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : secret "metrics-daemon-secret" not found
Sep 17 23:11:45.223442 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.223408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:45.225749 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.225728    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 23:11:45.236894 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.236865    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9ae1b512-2e62-465c-8d6f-8e782e948808-original-pull-secret\") pod \"global-pull-secret-syncer-d4lvj\" (UID: \"9ae1b512-2e62-465c-8d6f-8e782e948808\") " pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:45.324828 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.324784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:45.327801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.327783    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 23:11:45.337423 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.337396    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 23:11:45.347824 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.347793    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlc5n\" (UniqueName: \"kubernetes.io/projected/c31f4265-c4c3-49a8-b13a-e84f31c0f6f2-kube-api-access-dlc5n\") pod \"network-check-target-ttv8n\" (UID: \"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2\") " pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:45.427467 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.427395    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qw6rm\""
Sep 17 23:11:45.435603 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.435584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:45.438855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.438838    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4lvj"
Sep 17 23:11:45.626997 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.626967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:11:45.627203 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.627014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:11:45.627203 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:45.627127    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:11:45.627203 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:45.627188    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:17.627170048 +0000 UTC m=+98.721697077 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:11:45.627369 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:45.627130    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:11:45.627369 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:45.627228    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:11:45.627369 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:11:45.627279    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:12:17.627264652 +0000 UTC m=+98.721791666 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:11:45.723023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.722947    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ttv8n"]
Sep 17 23:11:45.726207 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:11:45.726186    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc31f4265_c4c3_49a8_b13a_e84f31c0f6f2.slice/crio-6e725c6b2dfafb5f34c10cb7f75e9ca7b30046c1190c2a24e8ed294efda8b66b WatchSource:0}: Error finding container 6e725c6b2dfafb5f34c10cb7f75e9ca7b30046c1190c2a24e8ed294efda8b66b: Status 404 returned error can't find the container with id 6e725c6b2dfafb5f34c10cb7f75e9ca7b30046c1190c2a24e8ed294efda8b66b
Sep 17 23:11:45.746224 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.746196    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d4lvj"]
Sep 17 23:11:45.749534 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:11:45.749514    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae1b512_2e62_465c_8d6f_8e782e948808.slice/crio-cd35f5579b87f6cf02569ea7d7a13a4dd907adf3892fff03597bb3f383c0ee04 WatchSource:0}: Error finding container cd35f5579b87f6cf02569ea7d7a13a4dd907adf3892fff03597bb3f383c0ee04: Status 404 returned error can't find the container with id cd35f5579b87f6cf02569ea7d7a13a4dd907adf3892fff03597bb3f383c0ee04
Sep 17 23:11:45.797260 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.797223    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttv8n" event={"ID":"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2","Type":"ContainerStarted","Data":"6e725c6b2dfafb5f34c10cb7f75e9ca7b30046c1190c2a24e8ed294efda8b66b"}
Sep 17 23:11:45.798255 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:45.798232    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d4lvj" event={"ID":"9ae1b512-2e62-465c-8d6f-8e782e948808","Type":"ContainerStarted","Data":"cd35f5579b87f6cf02569ea7d7a13a4dd907adf3892fff03597bb3f383c0ee04"}
Sep 17 23:11:50.813751 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:50.813709    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ttv8n" event={"ID":"c31f4265-c4c3-49a8-b13a-e84f31c0f6f2","Type":"ContainerStarted","Data":"f6b54758dd818ff6c506119abd89ee055f511fad229e47fa9e409883023a177d"}
Sep 17 23:11:50.814285 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:50.813815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:11:50.815118 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:50.815096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d4lvj" event={"ID":"9ae1b512-2e62-465c-8d6f-8e782e948808","Type":"ContainerStarted","Data":"fad19d0df249232a5bec81699ddffc73f9ef80574b9e5215bfae9faa9525e1e1"}
Sep 17 23:11:50.828373 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:50.828314    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ttv8n" podStartSLOduration=66.621268691 podStartE2EDuration="1m10.828295925s" totalImagesPullingTime="4.207027234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:50.827970356 +0000 UTC m=+71.922497393" watchObservedRunningTime="2026-09-17 23:11:50.828295925 +0000 UTC m=+71.922822961"
Sep 17 23:11:50.842740 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:11:50.842692    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d4lvj" podStartSLOduration=66.65368755 podStartE2EDuration="1m10.842674462s" totalImagesPullingTime="4.188986912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:11:50.84173107 +0000 UTC m=+71.936258105" watchObservedRunningTime="2026-09-17 23:11:50.842674462 +0000 UTC m=+71.937201498"
Sep 17 23:12:17.658558 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:12:17.658510    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:12:17.658985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:12:17.658575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:12:17.658985 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:12:17.658666    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:12:17.658985 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:12:17.658684    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:12:17.658985 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:12:17.658732    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:21.658718215 +0000 UTC m=+162.753245233 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:12:17.658985 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:12:17.658669    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:12:17.658985 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:12:17.658782    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:13:21.658771394 +0000 UTC m=+162.753298413 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:12:21.820896 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:12:21.820859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ttv8n"
Sep 17 23:12:49.186632 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:12:49.186576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:12:49.187069 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:12:49.186727    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 23:12:49.187069 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:12:49.186810    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs podName:94abe8fb-5a88-4039-884a-1397dec13323 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:51.18679344 +0000 UTC m=+252.281320454 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs") pod "network-metrics-daemon-mpl4d" (UID: "94abe8fb-5a88-4039-884a-1397dec13323") : secret "metrics-daemon-secret" not found
Sep 17 23:13:16.768841 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:16.768792    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8"
Sep 17 23:13:16.776955 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:16.776927    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nc7w8" podUID="fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5"
Sep 17 23:13:17.017722 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:17.017695    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:13:17.017880 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:17.017734    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:13:18.432799 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:18.432741    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mpl4d" podUID="94abe8fb-5a88-4039-884a-1397dec13323"
Sep 17 23:13:21.710706 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:21.710656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:13:21.710706 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:21.710711    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:13:21.711298 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:21.710812    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 23:13:21.711298 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:21.710814    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 23:13:21.711298 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:21.710918    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls podName:fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:23.710877331 +0000 UTC m=+284.805404364 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls") pod "dns-default-nc7w8" (UID: "fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5") : secret "dns-default-metrics-tls" not found
Sep 17 23:13:21.711298 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:21.710921    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5874477fdf-6xtqp: secret "image-registry-tls" not found
Sep 17 23:13:21.711298 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:13:21.710978    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls podName:d957c71e-eb0d-4976-97d7-21058a4171c8 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:23.710963577 +0000 UTC m=+284.805490591 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls") pod "image-registry-5874477fdf-6xtqp" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8") : secret "image-registry-tls" not found
Sep 17 23:13:22.047333 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.047252    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"]
Sep 17 23:13:22.048872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.048856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.052647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.052619    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 23:13:22.052647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.052630    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 23:13:22.052647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.052650    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 23:13:22.052886 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.052638    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 17 23:13:22.059626 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.059599    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"]
Sep 17 23:13:22.112370 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.112327    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69894b4f-2aed-4700-b082-a9193b5d32f3-tmp\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.112552 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.112426    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/69894b4f-2aed-4700-b082-a9193b5d32f3-klusterlet-config\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.112552 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.112447    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztj5q\" (UniqueName: \"kubernetes.io/projected/69894b4f-2aed-4700-b082-a9193b5d32f3-kube-api-access-ztj5q\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.213446 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.213404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/69894b4f-2aed-4700-b082-a9193b5d32f3-klusterlet-config\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.213446 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.213444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztj5q\" (UniqueName: \"kubernetes.io/projected/69894b4f-2aed-4700-b082-a9193b5d32f3-kube-api-access-ztj5q\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.213692 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.213472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69894b4f-2aed-4700-b082-a9193b5d32f3-tmp\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.213797 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.213780    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69894b4f-2aed-4700-b082-a9193b5d32f3-tmp\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.216004 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.215981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/69894b4f-2aed-4700-b082-a9193b5d32f3-klusterlet-config\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.223144 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.223118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztj5q\" (UniqueName: \"kubernetes.io/projected/69894b4f-2aed-4700-b082-a9193b5d32f3-kube-api-access-ztj5q\") pod \"klusterlet-addon-workmgr-674d7d766c-g5fhx\" (UID: \"69894b4f-2aed-4700-b082-a9193b5d32f3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.358039 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.357996    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:22.626717 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:22.626631    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"]
Sep 17 23:13:22.629564 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:13:22.629537    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69894b4f_2aed_4700_b082_a9193b5d32f3.slice/crio-d74790c21bf8de093e8385a21a6c95f18d9f10b4be54cc9a484b9cd74dc33d7a WatchSource:0}: Error finding container d74790c21bf8de093e8385a21a6c95f18d9f10b4be54cc9a484b9cd74dc33d7a: Status 404 returned error can't find the container with id d74790c21bf8de093e8385a21a6c95f18d9f10b4be54cc9a484b9cd74dc33d7a
Sep 17 23:13:23.032414 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:23.032324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx" event={"ID":"69894b4f-2aed-4700-b082-a9193b5d32f3","Type":"ContainerStarted","Data":"d74790c21bf8de093e8385a21a6c95f18d9f10b4be54cc9a484b9cd74dc33d7a"}
Sep 17 23:13:33.058385 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:33.058341    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx" event={"ID":"69894b4f-2aed-4700-b082-a9193b5d32f3","Type":"ContainerStarted","Data":"46723ca5c69221d38c69a23885d208372507723d153e106396d9317d2dff5c0e"}
Sep 17 23:13:33.058760 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:33.058560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:33.060246 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:33.060226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:13:33.078151 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:33.076198    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx" podStartSLOduration=1.58359662 podStartE2EDuration="11.076176812s" totalImagesPullingTime="9.492580192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:13:33.074227501 +0000 UTC m=+174.168754537" watchObservedRunningTime="2026-09-17 23:13:33.076176812 +0000 UTC m=+174.170703851"
Sep 17 23:13:33.413279 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:13:33.413250    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:14:30.083625 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.083590    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"]
Sep 17 23:14:30.086505 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.086487    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.089650 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.089621    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 17 23:14:30.089873 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.089855    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.090816 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.090792    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j7wlc\""
Sep 17 23:14:30.092308 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.092283    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.092429 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.092327    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 17 23:14:30.095716 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.095689    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tvnc7"]
Sep 17 23:14:30.098549 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.098532    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.101075 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.101057    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 23:14:30.101915 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.101884    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-clzj5\""
Sep 17 23:14:30.102929 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.102913    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 23:14:30.103263 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.103250    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.103334 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.103271    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.110060 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.110035    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 23:14:30.110626 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.110601    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"]
Sep 17 23:14:30.128505 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.128472    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tvnc7"]
Sep 17 23:14:30.189160 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f41801af-ae26-4324-8400-1ace17f3414d-tmp\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.189160 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/16844c76-db20-404d-b5ed-b7fcac6f49db-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.189419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189182    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.189419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189304    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41801af-ae26-4324-8400-1ace17f3414d-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.189419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189346    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f41801af-ae26-4324-8400-1ace17f3414d-snapshots\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.189419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189409    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfcg\" (UniqueName: \"kubernetes.io/projected/16844c76-db20-404d-b5ed-b7fcac6f49db-kube-api-access-dqfcg\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.189600 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189443    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f41801af-ae26-4324-8400-1ace17f3414d-serving-cert\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.189600 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41801af-ae26-4324-8400-1ace17f3414d-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.189600 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.189518    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tkcv\" (UniqueName: \"kubernetes.io/projected/f41801af-ae26-4324-8400-1ace17f3414d-kube-api-access-5tkcv\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.279504 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.279465    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"]
Sep 17 23:14:30.282389 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.282370    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"
Sep 17 23:14:30.284834 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.284810    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 23:14:30.284976 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.284876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:14:30.285101 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.285083    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-znz54\""
Sep 17 23:14:30.289418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.289394    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"]
Sep 17 23:14:30.289917 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.289871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfcg\" (UniqueName: \"kubernetes.io/projected/16844c76-db20-404d-b5ed-b7fcac6f49db-kube-api-access-dqfcg\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.290030 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.289954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f41801af-ae26-4324-8400-1ace17f3414d-serving-cert\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.290030 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41801af-ae26-4324-8400-1ace17f3414d-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.290145 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290033    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tkcv\" (UniqueName: \"kubernetes.io/projected/f41801af-ae26-4324-8400-1ace17f3414d-kube-api-access-5tkcv\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.290145 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f41801af-ae26-4324-8400-1ace17f3414d-tmp\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.290288 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290266    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/16844c76-db20-404d-b5ed-b7fcac6f49db-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.290352 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290328    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.290408 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41801af-ae26-4324-8400-1ace17f3414d-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.290459 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f41801af-ae26-4324-8400-1ace17f3414d-snapshots\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.290546 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.290520    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f41801af-ae26-4324-8400-1ace17f3414d-tmp\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.290705 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:30.290681    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:30.290817 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:30.290759    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls podName:16844c76-db20-404d-b5ed-b7fcac6f49db nodeName:}" failed. No retries permitted until 2026-09-17 23:14:30.790739685 +0000 UTC m=+231.885266699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-77mnh" (UID: "16844c76-db20-404d-b5ed-b7fcac6f49db") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:30.291052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.291028    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/16844c76-db20-404d-b5ed-b7fcac6f49db-telemetry-config\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.291187 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.291073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f41801af-ae26-4324-8400-1ace17f3414d-snapshots\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.291594 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.291572    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41801af-ae26-4324-8400-1ace17f3414d-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.291770 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.291753    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f41801af-ae26-4324-8400-1ace17f3414d-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.292735 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.292716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f41801af-ae26-4324-8400-1ace17f3414d-serving-cert\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.302487 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.302463    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfcg\" (UniqueName: \"kubernetes.io/projected/16844c76-db20-404d-b5ed-b7fcac6f49db-kube-api-access-dqfcg\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.302619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.302599    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tkcv\" (UniqueName: \"kubernetes.io/projected/f41801af-ae26-4324-8400-1ace17f3414d-kube-api-access-5tkcv\") pod \"insights-operator-6f9c644dd9-tvnc7\" (UID: \"f41801af-ae26-4324-8400-1ace17f3414d\") " pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.391707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.391613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62xfl\" (UniqueName: \"kubernetes.io/projected/4d41971a-8a8e-4003-8d1e-4aba3d5bc846-kube-api-access-62xfl\") pod \"volume-data-source-validator-5c4db8bdf9-qrjg6\" (UID: \"4d41971a-8a8e-4003-8d1e-4aba3d5bc846\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"
Sep 17 23:14:30.407780 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.407761    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7"
Sep 17 23:14:30.492532 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.492490    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62xfl\" (UniqueName: \"kubernetes.io/projected/4d41971a-8a8e-4003-8d1e-4aba3d5bc846-kube-api-access-62xfl\") pod \"volume-data-source-validator-5c4db8bdf9-qrjg6\" (UID: \"4d41971a-8a8e-4003-8d1e-4aba3d5bc846\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"
Sep 17 23:14:30.500670 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.500648    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62xfl\" (UniqueName: \"kubernetes.io/projected/4d41971a-8a8e-4003-8d1e-4aba3d5bc846-kube-api-access-62xfl\") pod \"volume-data-source-validator-5c4db8bdf9-qrjg6\" (UID: \"4d41971a-8a8e-4003-8d1e-4aba3d5bc846\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"
Sep 17 23:14:30.592808 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.592771    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"
Sep 17 23:14:30.675449 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.675357    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-tvnc7"]
Sep 17 23:14:30.677959 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:14:30.677935    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41801af_ae26_4324_8400_1ace17f3414d.slice/crio-16959a2813eea1114cf4e28b8df3a5f0ef077dad1d8058474858b073bdc6e0e8 WatchSource:0}: Error finding container 16959a2813eea1114cf4e28b8df3a5f0ef077dad1d8058474858b073bdc6e0e8: Status 404 returned error can't find the container with id 16959a2813eea1114cf4e28b8df3a5f0ef077dad1d8058474858b073bdc6e0e8
Sep 17 23:14:30.795106 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.795064    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:30.795276 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:30.795210    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:30.795276 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:30.795274    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls podName:16844c76-db20-404d-b5ed-b7fcac6f49db nodeName:}" failed. No retries permitted until 2026-09-17 23:14:31.795257102 +0000 UTC m=+232.889784127 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-77mnh" (UID: "16844c76-db20-404d-b5ed-b7fcac6f49db") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:30.859882 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:30.859847    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6"]
Sep 17 23:14:30.862998 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:14:30.862970    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d41971a_8a8e_4003_8d1e_4aba3d5bc846.slice/crio-b65e9fca0a44002aa8d7e93ff500402b9f4decb11244f2f4990544fb22de864f WatchSource:0}: Error finding container b65e9fca0a44002aa8d7e93ff500402b9f4decb11244f2f4990544fb22de864f: Status 404 returned error can't find the container with id b65e9fca0a44002aa8d7e93ff500402b9f4decb11244f2f4990544fb22de864f
Sep 17 23:14:31.200154 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:31.200115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7" event={"ID":"f41801af-ae26-4324-8400-1ace17f3414d","Type":"ContainerStarted","Data":"16959a2813eea1114cf4e28b8df3a5f0ef077dad1d8058474858b073bdc6e0e8"}
Sep 17 23:14:31.201373 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:31.201346    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6" event={"ID":"4d41971a-8a8e-4003-8d1e-4aba3d5bc846","Type":"ContainerStarted","Data":"b65e9fca0a44002aa8d7e93ff500402b9f4decb11244f2f4990544fb22de864f"}
Sep 17 23:14:31.803609 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:31.803563    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:31.803803 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:31.803736    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:31.803859 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:31.803820    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls podName:16844c76-db20-404d-b5ed-b7fcac6f49db nodeName:}" failed. No retries permitted until 2026-09-17 23:14:33.803798527 +0000 UTC m=+234.898325547 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-77mnh" (UID: "16844c76-db20-404d-b5ed-b7fcac6f49db") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:33.209445 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.209404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7" event={"ID":"f41801af-ae26-4324-8400-1ace17f3414d","Type":"ContainerStarted","Data":"5d39abfc9c404083cd47e8d157d570cb96a5f82bf66343f8b637903cb0375249"}
Sep 17 23:14:33.210959 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.210935    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6" event={"ID":"4d41971a-8a8e-4003-8d1e-4aba3d5bc846","Type":"ContainerStarted","Data":"5720c2e46ff94e5d2f32bc750bced585298598105b7f3ede93c78583999f278e"}
Sep 17 23:14:33.225654 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.225596    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7" podStartSLOduration=1.24823145 podStartE2EDuration="3.225578782s" totalImagesPullingTime="1.977347332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:33.224988233 +0000 UTC m=+234.319515269" watchObservedRunningTime="2026-09-17 23:14:33.225578782 +0000 UTC m=+234.320105817"
Sep 17 23:14:33.239968 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.239892    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-qrjg6" podStartSLOduration=1.4477889 podStartE2EDuration="3.239869752s" totalImagesPullingTime="1.792080852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:33.238811843 +0000 UTC m=+234.333338881" watchObservedRunningTime="2026-09-17 23:14:33.239869752 +0000 UTC m=+234.334396789"
Sep 17 23:14:33.766431 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.766391    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9crng"]
Sep 17 23:14:33.769741 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.769715    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:33.772695 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.772659    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 23:14:33.772855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.772836    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v429\""
Sep 17 23:14:33.773122 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.773107    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 23:14:33.781882 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.781848    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9crng"]
Sep 17 23:14:33.821708 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.821674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:33.821886 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:33.821823    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:33.821969 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:33.821895    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls podName:16844c76-db20-404d-b5ed-b7fcac6f49db nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.821878614 +0000 UTC m=+238.916405628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-77mnh" (UID: "16844c76-db20-404d-b5ed-b7fcac6f49db") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:33.922288 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.922253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwlr\" (UniqueName: \"kubernetes.io/projected/27628199-2db6-4d59-993b-b126afbe3304-kube-api-access-xzwlr\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:33.922452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.922306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:33.922452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.922331    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27628199-2db6-4d59-993b-b126afbe3304-data-volume\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:33.922452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.922366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:33.922452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:33.922424    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27628199-2db6-4d59-993b-b126afbe3304-crio-socket\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.023556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.023470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.023556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.023511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27628199-2db6-4d59-993b-b126afbe3304-data-volume\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.023556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.023528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.023556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.023546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27628199-2db6-4d59-993b-b126afbe3304-crio-socket\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.023829 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.023603    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:34.023829 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.023655    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:34.023829 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.023616    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwlr\" (UniqueName: \"kubernetes.io/projected/27628199-2db6-4d59-993b-b126afbe3304-kube-api-access-xzwlr\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.023829 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.023693    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:34.523673628 +0000 UTC m=+235.618200666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:34.023829 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.023709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/27628199-2db6-4d59-993b-b126afbe3304-crio-socket\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.023829 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.023715    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:34.523706493 +0000 UTC m=+235.618233507 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:34.024052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.023836    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/27628199-2db6-4d59-993b-b126afbe3304-data-volume\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.031276 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.031245    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwlr\" (UniqueName: \"kubernetes.io/projected/27628199-2db6-4d59-993b-b126afbe3304-kube-api-access-xzwlr\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.529505 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.529444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.529505 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:34.529503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:34.529981 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.529587    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:34.529981 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.529650    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:34.529981 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.529659    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:35.52964263 +0000 UTC m=+236.624169644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:34.529981 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:34.529692    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:35.529680598 +0000 UTC m=+236.624207612 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:35.279928 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.279881    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"]
Sep 17 23:14:35.282937 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.282920    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"
Sep 17 23:14:35.285345 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.285322    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 23:14:35.286319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.286304    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9hrg\""
Sep 17 23:14:35.286374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.286362    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 23:14:35.292260 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.292236    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"]
Sep 17 23:14:35.436010 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.435977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw62f\" (UniqueName: \"kubernetes.io/projected/ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7-kube-api-access-fw62f\") pod \"migrator-76794cbd6c-zqm4k\" (UID: \"ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"
Sep 17 23:14:35.523448 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.523421    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9ngsd_3963701d-1a30-4685-99f4-d9a1bc8af35b/dns-node-resolver/0.log"
Sep 17 23:14:35.536817 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.536734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw62f\" (UniqueName: \"kubernetes.io/projected/ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7-kube-api-access-fw62f\") pod \"migrator-76794cbd6c-zqm4k\" (UID: \"ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"
Sep 17 23:14:35.536817 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.536777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:35.536817 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.536799    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:35.537272 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:35.536884    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:35.537272 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:35.536915    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:35.537272 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:35.536982    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.53695913 +0000 UTC m=+238.631486146 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:35.537272 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:35.537019    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:37.53700545 +0000 UTC m=+238.631532464 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:35.544585 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.544558    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw62f\" (UniqueName: \"kubernetes.io/projected/ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7-kube-api-access-fw62f\") pod \"migrator-76794cbd6c-zqm4k\" (UID: \"ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"
Sep 17 23:14:35.592157 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.592115    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"
Sep 17 23:14:35.859934 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:35.859886    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k"]
Sep 17 23:14:35.863001 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:14:35.862978    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9b041d_da68_44ef_9c49_ea9a5cfd1dc7.slice/crio-581839a75187734934cc7232fa02168fe8b579cbe53afd8bcc06b896943c445f WatchSource:0}: Error finding container 581839a75187734934cc7232fa02168fe8b579cbe53afd8bcc06b896943c445f: Status 404 returned error can't find the container with id 581839a75187734934cc7232fa02168fe8b579cbe53afd8bcc06b896943c445f
Sep 17 23:14:36.220260 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:36.220183    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k" event={"ID":"ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7","Type":"ContainerStarted","Data":"581839a75187734934cc7232fa02168fe8b579cbe53afd8bcc06b896943c445f"}
Sep 17 23:14:36.723916 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:36.723871    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szjtn_021443d4-2c25-4789-9f74-acc54f1e1095/node-ca/0.log"
Sep 17 23:14:37.224682 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:37.224647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k" event={"ID":"ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7","Type":"ContainerStarted","Data":"6ce794ae74aab89863f3c46634a68e0f8b60722c254c077ebb000580dd2a9166"}
Sep 17 23:14:37.224857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:37.224702    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k" event={"ID":"ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7","Type":"ContainerStarted","Data":"bec4fa6332843505e26c1ebb5ea7624d13ca1a636dd5a5b7c57d72b33365adfb"}
Sep 17 23:14:37.239712 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:37.239652    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-zqm4k" podStartSLOduration=1.255567524 podStartE2EDuration="2.239634043s" totalImagesPullingTime="984.066519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:37.238629689 +0000 UTC m=+238.333156725" watchObservedRunningTime="2026-09-17 23:14:37.239634043 +0000 UTC m=+238.334161124"
Sep 17 23:14:37.552361 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:37.552262    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:37.552361 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:37.552319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:37.552549 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:37.552383    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:37.552549 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:37.552450    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:41.552431588 +0000 UTC m=+242.646958602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:37.552549 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:37.552451    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:37.552549 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:37.552495    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:41.552481259 +0000 UTC m=+242.647008273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:37.723557 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:37.723528    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8nq7_613efdc2-51ea-45ce-b410-861640d21bbf/serve-healthcheck-canary/0.log"
Sep 17 23:14:37.855144 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:37.855109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:37.855530 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:37.855260    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:37.855530 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:37.855324    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls podName:16844c76-db20-404d-b5ed-b7fcac6f49db nodeName:}" failed. No retries permitted until 2026-09-17 23:14:45.855308961 +0000 UTC m=+246.949835975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-77mnh" (UID: "16844c76-db20-404d-b5ed-b7fcac6f49db") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:38.435259 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.435225    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-nn8lg"]
Sep 17 23:14:38.438319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.438303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.441053 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.441035    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 17 23:14:38.441213 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.441195    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c729m\""
Sep 17 23:14:38.441299 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.441280    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 17 23:14:38.442193 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.442173    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 17 23:14:38.442316 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.442175    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 17 23:14:38.445228 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.445205    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-nn8lg"]
Sep 17 23:14:38.561248 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.561211    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cac1e42b-8e49-478c-8685-da3b4d6ed797-signing-cabundle\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.561414 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.561340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cac1e42b-8e49-478c-8685-da3b4d6ed797-signing-key\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.561414 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.561372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wq5\" (UniqueName: \"kubernetes.io/projected/cac1e42b-8e49-478c-8685-da3b4d6ed797-kube-api-access-75wq5\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.662008 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.661959    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cac1e42b-8e49-478c-8685-da3b4d6ed797-signing-key\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.662008 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.662004    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wq5\" (UniqueName: \"kubernetes.io/projected/cac1e42b-8e49-478c-8685-da3b4d6ed797-kube-api-access-75wq5\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.662257 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.662047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cac1e42b-8e49-478c-8685-da3b4d6ed797-signing-cabundle\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.662702 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.662683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cac1e42b-8e49-478c-8685-da3b4d6ed797-signing-cabundle\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.664472 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.664450    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cac1e42b-8e49-478c-8685-da3b4d6ed797-signing-key\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.670476 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.670451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wq5\" (UniqueName: \"kubernetes.io/projected/cac1e42b-8e49-478c-8685-da3b4d6ed797-kube-api-access-75wq5\") pod \"service-ca-55f6fc7b86-nn8lg\" (UID: \"cac1e42b-8e49-478c-8685-da3b4d6ed797\") " pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:38.747611 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:38.747526    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg"
Sep 17 23:14:39.016867 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:39.016790    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-55f6fc7b86-nn8lg"]
Sep 17 23:14:39.019414 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:14:39.019387    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac1e42b_8e49_478c_8685_da3b4d6ed797.slice/crio-cc523773d244334c29e26f9e822cba155e00750c32a247d11ccb7c02c6bcef72 WatchSource:0}: Error finding container cc523773d244334c29e26f9e822cba155e00750c32a247d11ccb7c02c6bcef72: Status 404 returned error can't find the container with id cc523773d244334c29e26f9e822cba155e00750c32a247d11ccb7c02c6bcef72
Sep 17 23:14:39.234762 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:39.234725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg" event={"ID":"cac1e42b-8e49-478c-8685-da3b4d6ed797","Type":"ContainerStarted","Data":"cc523773d244334c29e26f9e822cba155e00750c32a247d11ccb7c02c6bcef72"}
Sep 17 23:14:41.241545 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:41.241506    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg" event={"ID":"cac1e42b-8e49-478c-8685-da3b4d6ed797","Type":"ContainerStarted","Data":"8f0202f2bb9efd3843a3fa0cd5c45da6b1e98754de8de7e7a4c56e53ade9f039"}
Sep 17 23:14:41.260598 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:41.260537    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-55f6fc7b86-nn8lg" podStartSLOduration=1.251751652 podStartE2EDuration="3.260519013s" totalImagesPullingTime="2.008767361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:41.25937314 +0000 UTC m=+242.353900179" watchObservedRunningTime="2026-09-17 23:14:41.260519013 +0000 UTC m=+242.355046078"
Sep 17 23:14:41.584234 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:41.584197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:41.584404 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:41.584247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:41.584404 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:41.584309    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 23:14:41.584404 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:41.584379    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:49.584357174 +0000 UTC m=+250.678884201 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : configmap "kube-rbac-proxy" not found
Sep 17 23:14:41.584541 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:41.584408    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:41.584541 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:41.584465    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:14:49.584448807 +0000 UTC m=+250.678975840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:45.920364 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:45.920311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:14:45.920719 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:45.920472    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:45.920719 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:45.920538    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls podName:16844c76-db20-404d-b5ed-b7fcac6f49db nodeName:}" failed. No retries permitted until 2026-09-17 23:15:01.920522069 +0000 UTC m=+263.015049083 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-847b8bd8d7-77mnh" (UID: "16844c76-db20-404d-b5ed-b7fcac6f49db") : secret "cluster-monitoring-operator-tls" not found
Sep 17 23:14:49.648452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:49.648403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:49.649000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:49.648466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:49.649000 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:49.648611    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:49.649000 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:14:49.648691    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls podName:27628199-2db6-4d59-993b-b126afbe3304 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:05.648670252 +0000 UTC m=+266.743197283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9crng" (UID: "27628199-2db6-4d59-993b-b126afbe3304") : secret "insights-runtime-extractor-tls" not found
Sep 17 23:14:49.649135 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:49.649082    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/27628199-2db6-4d59-993b-b126afbe3304-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:14:51.258547 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:51.258505    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:14:51.260988 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:51.260968    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94abe8fb-5a88-4039-884a-1397dec13323-metrics-certs\") pod \"network-metrics-daemon-mpl4d\" (UID: \"94abe8fb-5a88-4039-884a-1397dec13323\") " pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:14:51.417209 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:51.417178    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mx9lw\""
Sep 17 23:14:51.425341 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:51.425318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mpl4d"
Sep 17 23:14:51.696737 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:51.696705    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mpl4d"]
Sep 17 23:14:51.699472 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:14:51.699444    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94abe8fb_5a88_4039_884a_1397dec13323.slice/crio-27480ede3f0b718fd29b776bbbb8d1859a7b6ae931ef334d04ec34eb673c130c WatchSource:0}: Error finding container 27480ede3f0b718fd29b776bbbb8d1859a7b6ae931ef334d04ec34eb673c130c: Status 404 returned error can't find the container with id 27480ede3f0b718fd29b776bbbb8d1859a7b6ae931ef334d04ec34eb673c130c
Sep 17 23:14:52.281195 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:52.281157    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpl4d" event={"ID":"94abe8fb-5a88-4039-884a-1397dec13323","Type":"ContainerStarted","Data":"27480ede3f0b718fd29b776bbbb8d1859a7b6ae931ef334d04ec34eb673c130c"}
Sep 17 23:14:53.286885 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:53.286804    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpl4d" event={"ID":"94abe8fb-5a88-4039-884a-1397dec13323","Type":"ContainerStarted","Data":"06b8e34dea1386138104871aa3fbed9c3fc7b673b669aa82a4e4834b5517e60a"}
Sep 17 23:14:53.286885 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:53.286854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mpl4d" event={"ID":"94abe8fb-5a88-4039-884a-1397dec13323","Type":"ContainerStarted","Data":"41ce29d7b98181292633c231321617af55b649997f965ad39ec210df9e6c25ef"}
Sep 17 23:14:53.304821 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:14:53.304770    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mpl4d" podStartSLOduration=252.138836235 podStartE2EDuration="4m13.304754703s" totalImagesPullingTime="1.165918468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:14:53.304578086 +0000 UTC m=+254.399105123" watchObservedRunningTime="2026-09-17 23:14:53.304754703 +0000 UTC m=+254.399281739"
Sep 17 23:15:01.937454 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:01.937352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:15:01.939948 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:01.939919    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/16844c76-db20-404d-b5ed-b7fcac6f49db-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-847b8bd8d7-77mnh\" (UID: \"16844c76-db20-404d-b5ed-b7fcac6f49db\") " pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:15:02.198490 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:02.198408    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j7wlc\""
Sep 17 23:15:02.206380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:02.206358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"
Sep 17 23:15:02.477992 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:02.477890    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh"]
Sep 17 23:15:02.480921 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:02.480881    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16844c76_db20_404d_b5ed_b7fcac6f49db.slice/crio-611482ae608dd3b8a5fb8fbb25799c8bbc4fde48c5b1800adb8dbbb802ead0fb WatchSource:0}: Error finding container 611482ae608dd3b8a5fb8fbb25799c8bbc4fde48c5b1800adb8dbbb802ead0fb: Status 404 returned error can't find the container with id 611482ae608dd3b8a5fb8fbb25799c8bbc4fde48c5b1800adb8dbbb802ead0fb
Sep 17 23:15:03.320524 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:03.320486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh" event={"ID":"16844c76-db20-404d-b5ed-b7fcac6f49db","Type":"ContainerStarted","Data":"611482ae608dd3b8a5fb8fbb25799c8bbc4fde48c5b1800adb8dbbb802ead0fb"}
Sep 17 23:15:04.893454 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:04.893416    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"]
Sep 17 23:15:04.896330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:04.896310    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"
Sep 17 23:15:04.898876 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:04.898839    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-hn5zl\""
Sep 17 23:15:04.898876 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:04.898853    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 23:15:04.905249 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:04.905222    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"]
Sep 17 23:15:04.964842 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:04.964799    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e155d60b-3aa5-47d1-a744-059cf5a07f73-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-vhm6n\" (UID: \"e155d60b-3aa5-47d1-a744-059cf5a07f73\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"
Sep 17 23:15:05.065783 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.065740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e155d60b-3aa5-47d1-a744-059cf5a07f73-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-vhm6n\" (UID: \"e155d60b-3aa5-47d1-a744-059cf5a07f73\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"
Sep 17 23:15:05.068253 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.068229    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e155d60b-3aa5-47d1-a744-059cf5a07f73-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-vhm6n\" (UID: \"e155d60b-3aa5-47d1-a744-059cf5a07f73\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"
Sep 17 23:15:05.206870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.206779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"
Sep 17 23:15:05.328862 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.328831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh" event={"ID":"16844c76-db20-404d-b5ed-b7fcac6f49db","Type":"ContainerStarted","Data":"7206d5c0395aa32c1c8aced93be378f7d93b14c388a89f588e0df2d2095af918"}
Sep 17 23:15:05.345565 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.345509    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-847b8bd8d7-77mnh" podStartSLOduration=33.417714656 podStartE2EDuration="35.345486476s" totalImagesPullingTime="1.92777182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:05.344444785 +0000 UTC m=+266.438971821" watchObservedRunningTime="2026-09-17 23:15:05.345486476 +0000 UTC m=+266.440013511"
Sep 17 23:15:05.476945 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.476847    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"]
Sep 17 23:15:05.479494 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:05.479460    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode155d60b_3aa5_47d1_a744_059cf5a07f73.slice/crio-edcc18c4e82a92946169f723b6a9780e645fa8e675ae449f5ab67621c67b179f WatchSource:0}: Error finding container edcc18c4e82a92946169f723b6a9780e645fa8e675ae449f5ab67621c67b179f: Status 404 returned error can't find the container with id edcc18c4e82a92946169f723b6a9780e645fa8e675ae449f5ab67621c67b179f
Sep 17 23:15:05.670506 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.670466    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:15:05.672890 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.672866    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/27628199-2db6-4d59-993b-b126afbe3304-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9crng\" (UID: \"27628199-2db6-4d59-993b-b126afbe3304\") " pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:15:05.881578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.881545    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9v429\""
Sep 17 23:15:05.889845 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:05.889813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9crng"
Sep 17 23:15:06.161875 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:06.161687    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9crng"]
Sep 17 23:15:06.164345 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:06.164316    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27628199_2db6_4d59_993b_b126afbe3304.slice/crio-646353eab0c4daff594c63483f9f6b406507052fb002a5af437a4e6a3f5f58c6 WatchSource:0}: Error finding container 646353eab0c4daff594c63483f9f6b406507052fb002a5af437a4e6a3f5f58c6: Status 404 returned error can't find the container with id 646353eab0c4daff594c63483f9f6b406507052fb002a5af437a4e6a3f5f58c6
Sep 17 23:15:06.334505 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:06.334471    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n" event={"ID":"e155d60b-3aa5-47d1-a744-059cf5a07f73","Type":"ContainerStarted","Data":"edcc18c4e82a92946169f723b6a9780e645fa8e675ae449f5ab67621c67b179f"}
Sep 17 23:15:06.336307 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:06.336262    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crng" event={"ID":"27628199-2db6-4d59-993b-b126afbe3304","Type":"ContainerStarted","Data":"4fd8ef1ecdeac79c2de734ff82bb285052619b457bdf3e4879b358c982fa6e38"}
Sep 17 23:15:06.336467 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:06.336335    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crng" event={"ID":"27628199-2db6-4d59-993b-b126afbe3304","Type":"ContainerStarted","Data":"646353eab0c4daff594c63483f9f6b406507052fb002a5af437a4e6a3f5f58c6"}
Sep 17 23:15:07.340760 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:07.340725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crng" event={"ID":"27628199-2db6-4d59-993b-b126afbe3304","Type":"ContainerStarted","Data":"f878e2500b19b458c7ab5f74128f5ac9d3e3bffc4fa490bd3e32349e4da2414f"}
Sep 17 23:15:07.342182 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:07.342158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n" event={"ID":"e155d60b-3aa5-47d1-a744-059cf5a07f73","Type":"ContainerStarted","Data":"10f1b019a238c1852ec55eff77487c7f85054c14fca353fbcc6e52854c5594ea"}
Sep 17 23:15:07.342394 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:07.342367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"
Sep 17 23:15:07.347166 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:07.347143    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n"
Sep 17 23:15:07.357510 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:07.357466    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-vhm6n" podStartSLOduration=2.119880437 podStartE2EDuration="3.357453341s" totalImagesPullingTime="1.237572904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:07.356585084 +0000 UTC m=+268.451112121" watchObservedRunningTime="2026-09-17 23:15:07.357453341 +0000 UTC m=+268.451980417"
Sep 17 23:15:09.349845 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:09.349808    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9crng" event={"ID":"27628199-2db6-4d59-993b-b126afbe3304","Type":"ContainerStarted","Data":"57663a1e1c93cb0429aa3557f6d10d325ea636f5a389a90400b4726a365bcd08"}
Sep 17 23:15:09.367177 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:09.367123    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9crng" podStartSLOduration=34.36088628 podStartE2EDuration="36.367106909s" totalImagesPullingTime="2.006220629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 23:14:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:09.366609281 +0000 UTC m=+270.461136331" watchObservedRunningTime="2026-09-17 23:15:09.367106909 +0000 UTC m=+270.461633979"
Sep 17 23:15:12.305673 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.305633    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mjjtn"]
Sep 17 23:15:12.309588 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.309557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.312006 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.311982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 23:15:12.312290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.312263    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 23:15:12.312412 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.312352    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 23:15:12.312520 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.312360    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bbsjl\""
Sep 17 23:15:12.313324 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.313310    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 23:15:12.421782 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.421736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-sys\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.421980 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.421792    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-textfile\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.421980 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.421917    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-wtmp\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.421980 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.421959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-tls\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.422231 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.422002    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.422231 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.422087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.422231 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.422121    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcbb2\" (UniqueName: \"kubernetes.io/projected/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-kube-api-access-xcbb2\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.422231 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.422179    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-metrics-client-ca\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.422231 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.422224    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-root\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.522686 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-root\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.522855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522740    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-sys\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.522855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522768    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-textfile\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.522855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522769    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-root\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.522855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-wtmp\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523094 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-tls\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523094 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522889    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523094 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523094 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcbb2\" (UniqueName: \"kubernetes.io/projected/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-kube-api-access-xcbb2\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523094 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.522990    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-metrics-client-ca\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523332 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.523167    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-textfile\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523332 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.523231    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-sys\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523787 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.523566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-metrics-client-ca\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.523787 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:15:12.523660    2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 23:15:12.523787 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:15:12.523711    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-tls podName:890bf2cd-29bf-4c8f-bcdd-474ec8455fc0 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:13.023694003 +0000 UTC m=+274.118221024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-tls") pod "node-exporter-mjjtn" (UID: "890bf2cd-29bf-4c8f-bcdd-474ec8455fc0") : secret "node-exporter-tls" not found
Sep 17 23:15:12.523787 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.523736    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.524039 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.523937    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-wtmp\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.526625 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.526264    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:12.532741 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:12.532685    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcbb2\" (UniqueName: \"kubernetes.io/projected/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-kube-api-access-xcbb2\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:13.028201 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.028166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-tls\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:13.030468 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.030429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/890bf2cd-29bf-4c8f-bcdd-474ec8455fc0-node-exporter-tls\") pod \"node-exporter-mjjtn\" (UID: \"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0\") " pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:13.221586 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.221551    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mjjtn"
Sep 17 23:15:13.233707 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:13.233671    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890bf2cd_29bf_4c8f_bcdd_474ec8455fc0.slice/crio-06e3a5e74c5d55b64d363fd17862c4709228cf8a2ad840f92a59569e9e6ce82d WatchSource:0}: Error finding container 06e3a5e74c5d55b64d363fd17862c4709228cf8a2ad840f92a59569e9e6ce82d: Status 404 returned error can't find the container with id 06e3a5e74c5d55b64d363fd17862c4709228cf8a2ad840f92a59569e9e6ce82d
Sep 17 23:15:13.365430 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.365395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjjtn" event={"ID":"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0","Type":"ContainerStarted","Data":"06e3a5e74c5d55b64d363fd17862c4709228cf8a2ad840f92a59569e9e6ce82d"}
Sep 17 23:15:13.404374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.404336    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:15:13.411595 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.411563    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.414419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.414388    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 23:15:13.414584 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.414392    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 23:15:13.414653 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.414409    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 23:15:13.414653 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.414626    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 23:15:13.414781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.414763    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 23:15:13.414846 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.414815    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fzb2d\""
Sep 17 23:15:13.414919 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.414862    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 23:15:13.415121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.415099    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 23:15:13.415203 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.415120    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 23:15:13.415203 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.415192    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 23:15:13.422598 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.422570    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:15:13.533585 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533547    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.533585 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533590    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.533853 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-out\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.533853 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533738    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.533853 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533763    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534014 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533907    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-web-config\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534014 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533951    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534014 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.533974    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.534045    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.534089    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9lt\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-kube-api-access-9n9lt\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.534142    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.534167    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.534163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.534193    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635150 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635150 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635390 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-out\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635390 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635258    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635390 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635545 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-web-config\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635773    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.635951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.635844    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.636847 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.636818    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.636975 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.636963    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.637428 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.637405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9lt\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-kube-api-access-9n9lt\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.637866 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.637840    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.638129 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.638102    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-out\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.638966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.638267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-volume\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.638966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.638280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.638966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.638486    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.638966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.638658    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.638966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.638844    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.638966 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.638925    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.639317 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.639128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.639810 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.639785    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-web-config\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.640469 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.640440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.640595 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.640578    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-tls-assets\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.640635 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.640587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.645940 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.645915    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9lt\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-kube-api-access-9n9lt\") pod \"alertmanager-main-0\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:13.723146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:13.723105    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:15:14.003831 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.003798    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:15:14.005159 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:14.005128    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc402b2e_6ba3_4f30_a24b_ac265bc74710.slice/crio-eab1d19ed869e7899b39ad56c3a492c25b6b502f1f99c53a8232981ea4367b18 WatchSource:0}: Error finding container eab1d19ed869e7899b39ad56c3a492c25b6b502f1f99c53a8232981ea4367b18: Status 404 returned error can't find the container with id eab1d19ed869e7899b39ad56c3a492c25b6b502f1f99c53a8232981ea4367b18
Sep 17 23:15:14.282166 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.282139    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"]
Sep 17 23:15:14.286822 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.286799    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.289411 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.289391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 23:15:14.289498 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.289448    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 23:15:14.289834 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.289798    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 23:15:14.290081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.290059    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 23:15:14.290741 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.290685    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 23:15:14.290946 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.290927    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-21u9vh7v7klr1\""
Sep 17 23:15:14.291040 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.290952    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-rcqr8\""
Sep 17 23:15:14.299808 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.299778    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"]
Sep 17 23:15:14.344698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.344663    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.344855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.344722    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.344855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.344744    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-metrics-client-ca\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.344855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.344801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-tls\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.344855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.344828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.345036 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.344943    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.345036 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.345021    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnf72\" (UniqueName: \"kubernetes.io/projected/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-kube-api-access-dnf72\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.345105 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.345053    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-grpc-tls\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.370265 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.370219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerStarted","Data":"eab1d19ed869e7899b39ad56c3a492c25b6b502f1f99c53a8232981ea4367b18"}
Sep 17 23:15:14.372081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.372055    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjjtn" event={"ID":"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0","Type":"ContainerStarted","Data":"dbd5a52b85c749f2d3ecbcdd0bce8f43e41056ee7fd9a8232cb4df7ec7df727a"}
Sep 17 23:15:14.446252 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.446215    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.446397 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.446257    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-metrics-client-ca\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.446397 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.446342    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-tls\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.446397 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.446380    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.446978 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.446950    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.447105 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.447017    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-metrics-client-ca\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.447105 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.447020    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnf72\" (UniqueName: \"kubernetes.io/projected/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-kube-api-access-dnf72\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.447105 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.447053    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-grpc-tls\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.447259 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.447106    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.449292 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.449261    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.449419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.449397    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.449462 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.449428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.449462 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.449429    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-grpc-tls\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.449539 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.449522    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-tls\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.449635 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.449619    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.455000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.454982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnf72\" (UniqueName: \"kubernetes.io/projected/fc275d57-fcd0-4671-91c8-6f0d91c9ac01-kube-api-access-dnf72\") pod \"thanos-querier-579d5ccdf6-rwbsq\" (UID: \"fc275d57-fcd0-4671-91c8-6f0d91c9ac01\") " pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.601041 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.601005    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:14.882726 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:14.882654    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"]
Sep 17 23:15:15.004775 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:15.004734    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc275d57_fcd0_4671_91c8_6f0d91c9ac01.slice/crio-11a5d98eb95df2a9047b99f924f8d9c945b4afeeb5c0d2c7f630638681f3de78 WatchSource:0}: Error finding container 11a5d98eb95df2a9047b99f924f8d9c945b4afeeb5c0d2c7f630638681f3de78: Status 404 returned error can't find the container with id 11a5d98eb95df2a9047b99f924f8d9c945b4afeeb5c0d2c7f630638681f3de78
Sep 17 23:15:15.376796 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:15.376759    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerID="40017622c7139f905853017748e87f95281f001cbb376fa7b144f7771d51bd91" exitCode=0
Sep 17 23:15:15.377240 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:15.376840    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"40017622c7139f905853017748e87f95281f001cbb376fa7b144f7771d51bd91"}
Sep 17 23:15:15.378176 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:15.378139    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" event={"ID":"fc275d57-fcd0-4671-91c8-6f0d91c9ac01","Type":"ContainerStarted","Data":"11a5d98eb95df2a9047b99f924f8d9c945b4afeeb5c0d2c7f630638681f3de78"}
Sep 17 23:15:15.379605 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:15.379581    2562 generic.go:356] "Generic (PLEG): container finished" podID="890bf2cd-29bf-4c8f-bcdd-474ec8455fc0" containerID="dbd5a52b85c749f2d3ecbcdd0bce8f43e41056ee7fd9a8232cb4df7ec7df727a" exitCode=0
Sep 17 23:15:15.379685 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:15.379661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjjtn" event={"ID":"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0","Type":"ContainerDied","Data":"dbd5a52b85c749f2d3ecbcdd0bce8f43e41056ee7fd9a8232cb4df7ec7df727a"}
Sep 17 23:15:16.386260 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:16.386212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjjtn" event={"ID":"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0","Type":"ContainerStarted","Data":"881b77212a331bf575b9ccccc8ab0238ccd0378e78109ec7b3e26e90b660a1f4"}
Sep 17 23:15:16.386729 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:16.386288    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mjjtn" event={"ID":"890bf2cd-29bf-4c8f-bcdd-474ec8455fc0","Type":"ContainerStarted","Data":"3acaca4a2466635d58cf1bd0b5b0be0c95e0e6cad8215fcb8362e97189192ea7"}
Sep 17 23:15:16.406671 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:16.406608    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mjjtn" podStartSLOduration=3.316429626 podStartE2EDuration="4.406591053s" totalImagesPullingTime="1.025190334s" totalInitContainerRuntime="64.971093ms" isStatefulPod=true podCreationTimestamp="2026-09-17 23:15:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:16.405262945 +0000 UTC m=+277.499789993" watchObservedRunningTime="2026-09-17 23:15:16.406591053 +0000 UTC m=+277.501118086"
Sep 17 23:15:17.071254 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.071212    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-94w87"]
Sep 17 23:15:17.074788 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.074761    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.077464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.077415    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8tmzx\""
Sep 17 23:15:17.077464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.077456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 23:15:17.083073 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.083014    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-94w87"]
Sep 17 23:15:17.174167 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.174125    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9cj6\" (UniqueName: \"kubernetes.io/projected/7a182c8b-3b46-447e-b0ea-2ea538079226-kube-api-access-d9cj6\") pod \"monitoring-plugin-5984459ff8-94w87\" (UID: \"7a182c8b-3b46-447e-b0ea-2ea538079226\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.174167 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.174173    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a182c8b-3b46-447e-b0ea-2ea538079226-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-94w87\" (UID: \"7a182c8b-3b46-447e-b0ea-2ea538079226\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.275083 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.275043    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9cj6\" (UniqueName: \"kubernetes.io/projected/7a182c8b-3b46-447e-b0ea-2ea538079226-kube-api-access-d9cj6\") pod \"monitoring-plugin-5984459ff8-94w87\" (UID: \"7a182c8b-3b46-447e-b0ea-2ea538079226\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.275083 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.275097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a182c8b-3b46-447e-b0ea-2ea538079226-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-94w87\" (UID: \"7a182c8b-3b46-447e-b0ea-2ea538079226\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.275275 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:15:17.275264    2562 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 23:15:17.275340 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:15:17.275330    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a182c8b-3b46-447e-b0ea-2ea538079226-monitoring-plugin-cert podName:7a182c8b-3b46-447e-b0ea-2ea538079226 nodeName:}" failed. No retries permitted until 2026-09-17 23:15:17.775311048 +0000 UTC m=+278.869838071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/7a182c8b-3b46-447e-b0ea-2ea538079226-monitoring-plugin-cert") pod "monitoring-plugin-5984459ff8-94w87" (UID: "7a182c8b-3b46-447e-b0ea-2ea538079226") : secret "monitoring-plugin-cert" not found
Sep 17 23:15:17.283603 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.283562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9cj6\" (UniqueName: \"kubernetes.io/projected/7a182c8b-3b46-447e-b0ea-2ea538079226-kube-api-access-d9cj6\") pod \"monitoring-plugin-5984459ff8-94w87\" (UID: \"7a182c8b-3b46-447e-b0ea-2ea538079226\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.394239 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.394206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerStarted","Data":"a4f62269dd819c33039f6705ea90393931a1c220f01ab659d123af27c8757a39"}
Sep 17 23:15:17.396636 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.396599    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" event={"ID":"fc275d57-fcd0-4671-91c8-6f0d91c9ac01","Type":"ContainerStarted","Data":"42a967135feb3d720f5bb3f02915ffd176f96da816ee41de2fda0c8899990ce8"}
Sep 17 23:15:17.780010 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.779973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a182c8b-3b46-447e-b0ea-2ea538079226-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-94w87\" (UID: \"7a182c8b-3b46-447e-b0ea-2ea538079226\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.782444 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.782420    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7a182c8b-3b46-447e-b0ea-2ea538079226-monitoring-plugin-cert\") pod \"monitoring-plugin-5984459ff8-94w87\" (UID: \"7a182c8b-3b46-447e-b0ea-2ea538079226\") " pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:17.987284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:17.987249    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:18.268817 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.268790    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5984459ff8-94w87"]
Sep 17 23:15:18.283962 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:18.283925    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a182c8b_3b46_447e_b0ea_2ea538079226.slice/crio-5c632fbd886c10c58af86b6e8dfa7ce8c79b6fdeb0859a63355af665ba3168d1 WatchSource:0}: Error finding container 5c632fbd886c10c58af86b6e8dfa7ce8c79b6fdeb0859a63355af665ba3168d1: Status 404 returned error can't find the container with id 5c632fbd886c10c58af86b6e8dfa7ce8c79b6fdeb0859a63355af665ba3168d1
Sep 17 23:15:18.408007 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.407969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerStarted","Data":"ae071b436a785956f609daca21876c83e78a5a4dd1251c4955d926613c00c54e"}
Sep 17 23:15:18.408402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.408061    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerStarted","Data":"fa169db794362201d834bb0ae9dd0136c6c6206c3e562b15f9f6e6ca093b9ade"}
Sep 17 23:15:18.408402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.408093    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerStarted","Data":"59b8fa231afbdd9206a65e2124d37fa5a874392b74ebd65c72f733da2145e0eb"}
Sep 17 23:15:18.408402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.408117    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerStarted","Data":"303608bd729c8a9ba917f8d62e5ee881b64d817b94f90fcd17bcf7e71c473bc0"}
Sep 17 23:15:18.408402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.408142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerStarted","Data":"bae58a33fb6af10dc29a6f1afea63f49c9dafbbe0b2ad8b500006852432ee484"}
Sep 17 23:15:18.410811 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.410786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" event={"ID":"fc275d57-fcd0-4671-91c8-6f0d91c9ac01","Type":"ContainerStarted","Data":"7c632438a1c6b502ee2a19590a9ea08b5fff01c6cb7c5592993368b9145c2a89"}
Sep 17 23:15:18.410941 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.410857    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" event={"ID":"fc275d57-fcd0-4671-91c8-6f0d91c9ac01","Type":"ContainerStarted","Data":"b17aaf48117d79a540e99d8572de60eece3988ee6ab0c85e7ebabe0272d33cee"}
Sep 17 23:15:18.410941 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.410887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" event={"ID":"fc275d57-fcd0-4671-91c8-6f0d91c9ac01","Type":"ContainerStarted","Data":"a634fc42b9b010b19cfda8a173313322c55fed83bdb2d94e479bdcd06d79101b"}
Sep 17 23:15:18.411876 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.411853    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87" event={"ID":"7a182c8b-3b46-447e-b0ea-2ea538079226","Type":"ContainerStarted","Data":"5c632fbd886c10c58af86b6e8dfa7ce8c79b6fdeb0859a63355af665ba3168d1"}
Sep 17 23:15:18.440878 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:18.440832    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.77718152 podStartE2EDuration="5.440813555s" totalImagesPullingTime="3.642536935s" totalInitContainerRuntime="21.0951ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:18.438573613 +0000 UTC m=+279.533100651" watchObservedRunningTime="2026-09-17 23:15:18.440813555 +0000 UTC m=+279.535340591"
Sep 17 23:15:19.421066 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:19.421029    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" event={"ID":"fc275d57-fcd0-4671-91c8-6f0d91c9ac01","Type":"ContainerStarted","Data":"559c31a5e9f73747b962717cc797a0b38e6af5e4048ae3699879d33b3daba9b6"}
Sep 17 23:15:19.421551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:19.421109    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" event={"ID":"fc275d57-fcd0-4671-91c8-6f0d91c9ac01","Type":"ContainerStarted","Data":"8e4156d795678852607cd6613a118e73728f1c014e9ea11a7b2b68f1de97b744"}
Sep 17 23:15:19.421551 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:19.421491    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:19.447023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:19.446962    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq" podStartSLOduration=2.360635778 podStartE2EDuration="5.446944178s" totalImagesPullingTime="3.0863084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:19.442353856 +0000 UTC m=+280.536880918" watchObservedRunningTime="2026-09-17 23:15:19.446944178 +0000 UTC m=+280.541471214"
Sep 17 23:15:20.019201 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:15:20.019144    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8"
Sep 17 23:15:20.019201 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:15:20.019144    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-nc7w8" podUID="fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5"
Sep 17 23:15:20.426819 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:20.426786    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:15:20.426819 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:20.426800    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:15:20.426819 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:20.426801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87" event={"ID":"7a182c8b-3b46-447e-b0ea-2ea538079226","Type":"ContainerStarted","Data":"30066816cda9e64fc9635fd4c3b2e67242eefef71958070249a1349230b6f0cf"}
Sep 17 23:15:20.427402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:20.427055    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:20.431958 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:20.431931    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87"
Sep 17 23:15:20.457266 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:20.457206    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5984459ff8-94w87" podStartSLOduration=2.201650248 podStartE2EDuration="3.457187926s" totalImagesPullingTime="1.255537678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:20.44211911 +0000 UTC m=+281.536646147" watchObservedRunningTime="2026-09-17 23:15:20.457187926 +0000 UTC m=+281.551714962"
Sep 17 23:15:23.736245 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:23.736186    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:15:23.736735 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:23.736283    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:15:23.738654 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:23.738618    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"image-registry-5874477fdf-6xtqp\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") " pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:15:23.738654 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:23.738631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5-metrics-tls\") pod \"dns-default-nc7w8\" (UID: \"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5\") " pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:15:24.033068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.032970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9dbhz\""
Sep 17 23:15:24.033068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.032970    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cvr9f\""
Sep 17 23:15:24.038124 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.038098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:15:24.038239 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.038160    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:15:24.321529 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.321504    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nc7w8"]
Sep 17 23:15:24.324473 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:24.324443    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe828f4_79f3_4e4a_b2a7_eaff07c8fee5.slice/crio-119ddd8e2d4a275981598f5ea5b79efd1fa5e43a9b1c334565a317cd6d2d16e7 WatchSource:0}: Error finding container 119ddd8e2d4a275981598f5ea5b79efd1fa5e43a9b1c334565a317cd6d2d16e7: Status 404 returned error can't find the container with id 119ddd8e2d4a275981598f5ea5b79efd1fa5e43a9b1c334565a317cd6d2d16e7
Sep 17 23:15:24.325100 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.325079    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5874477fdf-6xtqp"]
Sep 17 23:15:24.327245 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:24.327224    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd957c71e_eb0d_4976_97d7_21058a4171c8.slice/crio-4489af0886e0b0d3d412b7e36da3095fa5ae5fea910abcbc045471453fba8834 WatchSource:0}: Error finding container 4489af0886e0b0d3d412b7e36da3095fa5ae5fea910abcbc045471453fba8834: Status 404 returned error can't find the container with id 4489af0886e0b0d3d412b7e36da3095fa5ae5fea910abcbc045471453fba8834
Sep 17 23:15:24.442464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.442432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nc7w8" event={"ID":"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5","Type":"ContainerStarted","Data":"119ddd8e2d4a275981598f5ea5b79efd1fa5e43a9b1c334565a317cd6d2d16e7"}
Sep 17 23:15:24.443892 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.443866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" event={"ID":"d957c71e-eb0d-4976-97d7-21058a4171c8","Type":"ContainerStarted","Data":"5ec6f1076ac260a55fbd52e952f42285713a1b94261d03761bd772aadf4d1db3"}
Sep 17 23:15:24.444048 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.443926    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" event={"ID":"d957c71e-eb0d-4976-97d7-21058a4171c8","Type":"ContainerStarted","Data":"4489af0886e0b0d3d412b7e36da3095fa5ae5fea910abcbc045471453fba8834"}
Sep 17 23:15:24.444048 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.444000    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:15:24.464144 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:24.464090    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" podStartSLOduration=284.464071183 podStartE2EDuration="4m44.464071183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:24.463581847 +0000 UTC m=+285.558108883" watchObservedRunningTime="2026-09-17 23:15:24.464071183 +0000 UTC m=+285.558598222"
Sep 17 23:15:25.434870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.434838    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-579d5ccdf6-rwbsq"
Sep 17 23:15:25.687215 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.684930    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-gmqx7"]
Sep 17 23:15:25.689807 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.689778    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-gmqx7"
Sep 17 23:15:25.692374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.692346    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 23:15:25.692605 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.692587    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 23:15:25.692722 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.692663    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-nb8x5\""
Sep 17 23:15:25.695829 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.695803    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-gmqx7"]
Sep 17 23:15:25.857609 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.857560    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xff6\" (UniqueName: \"kubernetes.io/projected/e955fcd2-2eff-4492-87db-c4f4bb1312a2-kube-api-access-2xff6\") pod \"downloads-6594f4c75-gmqx7\" (UID: \"e955fcd2-2eff-4492-87db-c4f4bb1312a2\") " pod="openshift-console/downloads-6594f4c75-gmqx7"
Sep 17 23:15:25.959015 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.958979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xff6\" (UniqueName: \"kubernetes.io/projected/e955fcd2-2eff-4492-87db-c4f4bb1312a2-kube-api-access-2xff6\") pod \"downloads-6594f4c75-gmqx7\" (UID: \"e955fcd2-2eff-4492-87db-c4f4bb1312a2\") " pod="openshift-console/downloads-6594f4c75-gmqx7"
Sep 17 23:15:25.974355 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:25.974318    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xff6\" (UniqueName: \"kubernetes.io/projected/e955fcd2-2eff-4492-87db-c4f4bb1312a2-kube-api-access-2xff6\") pod \"downloads-6594f4c75-gmqx7\" (UID: \"e955fcd2-2eff-4492-87db-c4f4bb1312a2\") " pod="openshift-console/downloads-6594f4c75-gmqx7"
Sep 17 23:15:26.002724 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:26.002695    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-gmqx7"
Sep 17 23:15:26.288071 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:26.288046    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-gmqx7"]
Sep 17 23:15:26.290722 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:26.290696    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode955fcd2_2eff_4492_87db_c4f4bb1312a2.slice/crio-c4892846d7ca8c35de7a8b11bfe5836b6c644665bb871e8fc68ba92a19f6541b WatchSource:0}: Error finding container c4892846d7ca8c35de7a8b11bfe5836b6c644665bb871e8fc68ba92a19f6541b: Status 404 returned error can't find the container with id c4892846d7ca8c35de7a8b11bfe5836b6c644665bb871e8fc68ba92a19f6541b
Sep 17 23:15:26.453281 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:26.453243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nc7w8" event={"ID":"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5","Type":"ContainerStarted","Data":"e685b68001001d1dd3683aa79a726a412e4a01fcbd1eeaf5e11a393aedd20e52"}
Sep 17 23:15:26.453726 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:26.453297    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nc7w8" event={"ID":"fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5","Type":"ContainerStarted","Data":"9a449a6536a2be84684b90ad6923a2a9fa3e4b5fd8e0946be6f3429291b6f1ea"}
Sep 17 23:15:26.453726 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:26.453347    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:15:26.454496 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:26.454467    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-gmqx7" event={"ID":"e955fcd2-2eff-4492-87db-c4f4bb1312a2","Type":"ContainerStarted","Data":"c4892846d7ca8c35de7a8b11bfe5836b6c644665bb871e8fc68ba92a19f6541b"}
Sep 17 23:15:32.358955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:32.358880    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx" podUID="69894b4f-2aed-4700-b082-a9193b5d32f3" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/healthz\": dial tcp 10.133.0.9:8000: connect: connection refused"
Sep 17 23:15:32.483205 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:32.483118    2562 generic.go:356] "Generic (PLEG): container finished" podID="69894b4f-2aed-4700-b082-a9193b5d32f3" containerID="46723ca5c69221d38c69a23885d208372507723d153e106396d9317d2dff5c0e" exitCode=1
Sep 17 23:15:32.483372 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:32.483196    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx" event={"ID":"69894b4f-2aed-4700-b082-a9193b5d32f3","Type":"ContainerDied","Data":"46723ca5c69221d38c69a23885d208372507723d153e106396d9317d2dff5c0e"}
Sep 17 23:15:32.483628 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:32.483608    2562 scope.go:117] "RemoveContainer" containerID="46723ca5c69221d38c69a23885d208372507723d153e106396d9317d2dff5c0e"
Sep 17 23:15:32.498104 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:32.498047    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-nc7w8" podStartSLOduration=257.889051873 podStartE2EDuration="4m19.49802542s" totalImagesPullingTime="1.608973547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:26.470072114 +0000 UTC m=+287.564599151" watchObservedRunningTime="2026-09-17 23:15:32.49802542 +0000 UTC m=+293.592552449"
Sep 17 23:15:33.059210 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:33.059179    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:15:33.490603 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:33.490566    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx" event={"ID":"69894b4f-2aed-4700-b082-a9193b5d32f3","Type":"ContainerStarted","Data":"c04a508797181d83a4b69bf632158b18abf6de7a66582613c2d1205a4298f9d0"}
Sep 17 23:15:33.491107 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:33.490804    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:15:33.492033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:33.492009    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-674d7d766c-g5fhx"
Sep 17 23:15:35.394261 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.394225    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86868d784b-ldsd9"]
Sep 17 23:15:35.400758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.400730    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.403514 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.403485    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:15:35.403514 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.403492    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hjfdq\""
Sep 17 23:15:35.405614 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.405589    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:15:35.405829 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.405812    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:15:35.405925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.405885    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:15:35.405984 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.405929    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:15:35.413430 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.411022    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86868d784b-ldsd9"]
Sep 17 23:15:35.443317 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.443276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-service-ca\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.443317 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.443319    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxrrp\" (UniqueName: \"kubernetes.io/projected/980d629d-f13e-4e65-bc9e-ee128658e7e5-kube-api-access-pxrrp\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.443574 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.443352    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-serving-cert\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.443574 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.443375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-oauth-serving-cert\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.443574 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.443461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-oauth-config\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.443695 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.443579    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-config\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.544506 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.544461    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-oauth-config\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.544707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.544571    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-config\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.544707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.544605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-service-ca\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.544707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.544639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxrrp\" (UniqueName: \"kubernetes.io/projected/980d629d-f13e-4e65-bc9e-ee128658e7e5-kube-api-access-pxrrp\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.544707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.544665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-serving-cert\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.544707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.544681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-oauth-serving-cert\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.545530 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.545494    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-oauth-serving-cert\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.545800 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.545752    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-service-ca\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.545940 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.545893    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-config\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.548153 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.548129    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-serving-cert\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.548153 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.548138    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-oauth-config\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.553877 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.553851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxrrp\" (UniqueName: \"kubernetes.io/projected/980d629d-f13e-4e65-bc9e-ee128658e7e5-kube-api-access-pxrrp\") pod \"console-86868d784b-ldsd9\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") " pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:35.715771 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:35.715662    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:15:36.461589 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:36.461555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nc7w8"
Sep 17 23:15:42.411785 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:42.411754    2562 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 23:15:42.644272 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:42.644239    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86868d784b-ldsd9"]
Sep 17 23:15:42.646996 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:15:42.646958    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980d629d_f13e_4e65_bc9e_ee128658e7e5.slice/crio-7d97eb12c362a3298104175213975edd99d5068b12abc9b5bf396294b331871f WatchSource:0}: Error finding container 7d97eb12c362a3298104175213975edd99d5068b12abc9b5bf396294b331871f: Status 404 returned error can't find the container with id 7d97eb12c362a3298104175213975edd99d5068b12abc9b5bf396294b331871f
Sep 17 23:15:42.650103 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:42.649822    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:15:43.538927 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.538193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-gmqx7" event={"ID":"e955fcd2-2eff-4492-87db-c4f4bb1312a2","Type":"ContainerStarted","Data":"eb1056f1e9f21ba63c2b3fde69d1a6ec5c6598c943f0485dc8f4c2c9c86e171b"}
Sep 17 23:15:43.538927 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.538729    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-gmqx7"
Sep 17 23:15:43.542236 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.541882    2562 generic.go:356] "Generic (PLEG): container finished" podID="f41801af-ae26-4324-8400-1ace17f3414d" containerID="5d39abfc9c404083cd47e8d157d570cb96a5f82bf66343f8b637903cb0375249" exitCode=0
Sep 17 23:15:43.542236 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.541986    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7" event={"ID":"f41801af-ae26-4324-8400-1ace17f3414d","Type":"ContainerDied","Data":"5d39abfc9c404083cd47e8d157d570cb96a5f82bf66343f8b637903cb0375249"}
Sep 17 23:15:43.542609 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.542388    2562 scope.go:117] "RemoveContainer" containerID="5d39abfc9c404083cd47e8d157d570cb96a5f82bf66343f8b637903cb0375249"
Sep 17 23:15:43.545182 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.545131    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-gmqx7"
Sep 17 23:15:43.545697 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.545622    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86868d784b-ldsd9" event={"ID":"980d629d-f13e-4e65-bc9e-ee128658e7e5","Type":"ContainerStarted","Data":"7d97eb12c362a3298104175213975edd99d5068b12abc9b5bf396294b331871f"}
Sep 17 23:15:43.561314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:43.560607    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-gmqx7" podStartSLOduration=2.382082212 podStartE2EDuration="18.560583298s" totalImagesPullingTime="16.178501086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:43.556013438 +0000 UTC m=+304.650540530" watchObservedRunningTime="2026-09-17 23:15:43.560583298 +0000 UTC m=+304.655110337"
Sep 17 23:15:44.047179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:44.046795    2562 patch_prober.go:28] interesting pod/image-registry-5874477fdf-6xtqp container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 23:15:44.047179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:44.046887    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 23:15:44.555473 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:44.554977    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-tvnc7" event={"ID":"f41801af-ae26-4324-8400-1ace17f3414d","Type":"ContainerStarted","Data":"ce40a85298bfe0a69a70b5eefca1c1c82dae0761d4ab95d6f88a64fc26733750"}
Sep 17 23:15:45.456366 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:45.455424    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:15:49.585141 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:49.585096    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86868d784b-ldsd9" event={"ID":"980d629d-f13e-4e65-bc9e-ee128658e7e5","Type":"ContainerStarted","Data":"f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9"}
Sep 17 23:15:49.602881 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:49.602814    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86868d784b-ldsd9" podStartSLOduration=8.42815425 podStartE2EDuration="14.602791725s" totalImagesPullingTime="6.174637475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:15:49.602100842 +0000 UTC m=+310.696627878" watchObservedRunningTime="2026-09-17 23:15:49.602791725 +0000 UTC m=+310.697318762"
Sep 17 23:15:50.005938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:50.005835    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5874477fdf-6xtqp"]
Sep 17 23:15:53.857732 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:53.857695    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86868d784b-ldsd9"]
Sep 17 23:15:55.716110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:15:55.716072    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:16:15.033396 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:15.033325    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8" containerName="registry" containerID="cri-o://5ec6f1076ac260a55fbd52e952f42285713a1b94261d03761bd772aadf4d1db3" gracePeriod=30
Sep 17 23:16:15.448274 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:15.448239    2562 patch_prober.go:28] interesting pod/image-registry-5874477fdf-6xtqp container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused" start-of-body=
Sep 17 23:16:15.448441 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:15.448302    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8" containerName="registry" probeResult="failure" output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused"
Sep 17 23:16:16.698482 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.698431    2562 generic.go:356] "Generic (PLEG): container finished" podID="d957c71e-eb0d-4976-97d7-21058a4171c8" containerID="5ec6f1076ac260a55fbd52e952f42285713a1b94261d03761bd772aadf4d1db3" exitCode=0
Sep 17 23:16:16.699011 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.698486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" event={"ID":"d957c71e-eb0d-4976-97d7-21058a4171c8","Type":"ContainerDied","Data":"5ec6f1076ac260a55fbd52e952f42285713a1b94261d03761bd772aadf4d1db3"}
Sep 17 23:16:16.818857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.818822    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:16:16.901406 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.901379    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8jgw\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-kube-api-access-z8jgw\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.901573 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.901416    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-bound-sa-token\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.902134 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.901602    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-installation-pull-secrets\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.904606    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-trusted-ca\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.904612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-kube-api-access-z8jgw" (OuterVolumeSpecName: "kube-api-access-z8jgw") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "kube-api-access-z8jgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.904652    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d957c71e-eb0d-4976-97d7-21058a4171c8-ca-trust-extracted\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.904684    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-certificates\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.904761    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.904788    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-image-registry-private-configuration\") pod \"d957c71e-eb0d-4976-97d7-21058a4171c8\" (UID: \"d957c71e-eb0d-4976-97d7-21058a4171c8\") "
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.904855    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.905121    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8jgw\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-kube-api-access-z8jgw\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.905138    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-installation-pull-secrets\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.905171    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:16.908393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.905561    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:16.909039 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.908709    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:16.909039 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.908961    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:16.909728 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.909698    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:16.915813 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:16.915788    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d957c71e-eb0d-4976-97d7-21058a4171c8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d957c71e-eb0d-4976-97d7-21058a4171c8" (UID: "d957c71e-eb0d-4976-97d7-21058a4171c8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:16:17.006284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.006198    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-bound-sa-token\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:17.006284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.006230    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-trusted-ca\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:17.006284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.006239    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d957c71e-eb0d-4976-97d7-21058a4171c8-ca-trust-extracted\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:17.006284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.006252    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-certificates\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:17.006284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.006261    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d957c71e-eb0d-4976-97d7-21058a4171c8-registry-tls\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:17.006284 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.006271    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d957c71e-eb0d-4976-97d7-21058a4171c8-image-registry-private-configuration\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:17.703673 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.703596    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp"
Sep 17 23:16:17.703673 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.703613    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5874477fdf-6xtqp" event={"ID":"d957c71e-eb0d-4976-97d7-21058a4171c8","Type":"ContainerDied","Data":"4489af0886e0b0d3d412b7e36da3095fa5ae5fea910abcbc045471453fba8834"}
Sep 17 23:16:17.703673 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.703664    2562 scope.go:117] "RemoveContainer" containerID="5ec6f1076ac260a55fbd52e952f42285713a1b94261d03761bd772aadf4d1db3"
Sep 17 23:16:17.720009 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.719982    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5874477fdf-6xtqp"]
Sep 17 23:16:17.723942 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:17.723920    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5874477fdf-6xtqp"]
Sep 17 23:16:18.876687 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:18.876626    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-86868d784b-ldsd9" podUID="980d629d-f13e-4e65-bc9e-ee128658e7e5" containerName="console" containerID="cri-o://f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9" gracePeriod=15
Sep 17 23:16:19.156588 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.156558    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:16:19.322896 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.322854    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-config\") pod \"980d629d-f13e-4e65-bc9e-ee128658e7e5\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") "
Sep 17 23:16:19.322896 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.322923    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-oauth-serving-cert\") pod \"980d629d-f13e-4e65-bc9e-ee128658e7e5\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") "
Sep 17 23:16:19.323128 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.322945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-oauth-config\") pod \"980d629d-f13e-4e65-bc9e-ee128658e7e5\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") "
Sep 17 23:16:19.323128 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.322961    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-serving-cert\") pod \"980d629d-f13e-4e65-bc9e-ee128658e7e5\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") "
Sep 17 23:16:19.323128 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.322982    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-service-ca\") pod \"980d629d-f13e-4e65-bc9e-ee128658e7e5\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") "
Sep 17 23:16:19.323128 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.322998    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxrrp\" (UniqueName: \"kubernetes.io/projected/980d629d-f13e-4e65-bc9e-ee128658e7e5-kube-api-access-pxrrp\") pod \"980d629d-f13e-4e65-bc9e-ee128658e7e5\" (UID: \"980d629d-f13e-4e65-bc9e-ee128658e7e5\") "
Sep 17 23:16:19.323373 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.323347    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-config" (OuterVolumeSpecName: "console-config") pod "980d629d-f13e-4e65-bc9e-ee128658e7e5" (UID: "980d629d-f13e-4e65-bc9e-ee128658e7e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:19.323438 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.323421    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-service-ca" (OuterVolumeSpecName: "service-ca") pod "980d629d-f13e-4e65-bc9e-ee128658e7e5" (UID: "980d629d-f13e-4e65-bc9e-ee128658e7e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:19.323487 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.323442    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "980d629d-f13e-4e65-bc9e-ee128658e7e5" (UID: "980d629d-f13e-4e65-bc9e-ee128658e7e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:19.325296 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.325274    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980d629d-f13e-4e65-bc9e-ee128658e7e5-kube-api-access-pxrrp" (OuterVolumeSpecName: "kube-api-access-pxrrp") pod "980d629d-f13e-4e65-bc9e-ee128658e7e5" (UID: "980d629d-f13e-4e65-bc9e-ee128658e7e5"). InnerVolumeSpecName "kube-api-access-pxrrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:19.325436 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.325302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "980d629d-f13e-4e65-bc9e-ee128658e7e5" (UID: "980d629d-f13e-4e65-bc9e-ee128658e7e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:19.325436 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.325350    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "980d629d-f13e-4e65-bc9e-ee128658e7e5" (UID: "980d629d-f13e-4e65-bc9e-ee128658e7e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:19.418085 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.418006    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8" path="/var/lib/kubelet/pods/d957c71e-eb0d-4976-97d7-21058a4171c8/volumes"
Sep 17 23:16:19.423669 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.423647    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-config\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:19.423669 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.423665    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-oauth-serving-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:19.423844 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.423675    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-oauth-config\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:19.423844 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.423685    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/980d629d-f13e-4e65-bc9e-ee128658e7e5-console-serving-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:19.423844 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.423694    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/980d629d-f13e-4e65-bc9e-ee128658e7e5-service-ca\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:19.423844 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.423702    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxrrp\" (UniqueName: \"kubernetes.io/projected/980d629d-f13e-4e65-bc9e-ee128658e7e5-kube-api-access-pxrrp\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:19.714434 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.714339    2562 generic.go:356] "Generic (PLEG): container finished" podID="980d629d-f13e-4e65-bc9e-ee128658e7e5" containerID="f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9" exitCode=0
Sep 17 23:16:19.714434 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.714380    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86868d784b-ldsd9" event={"ID":"980d629d-f13e-4e65-bc9e-ee128658e7e5","Type":"ContainerDied","Data":"f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9"}
Sep 17 23:16:19.714434 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.714402    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86868d784b-ldsd9"
Sep 17 23:16:19.714698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.714445    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86868d784b-ldsd9" event={"ID":"980d629d-f13e-4e65-bc9e-ee128658e7e5","Type":"ContainerDied","Data":"7d97eb12c362a3298104175213975edd99d5068b12abc9b5bf396294b331871f"}
Sep 17 23:16:19.714698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.714466    2562 scope.go:117] "RemoveContainer" containerID="f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9"
Sep 17 23:16:19.723161 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.723141    2562 scope.go:117] "RemoveContainer" containerID="f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9"
Sep 17 23:16:19.723458 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:16:19.723439    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9\": container with ID starting with f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9 not found: ID does not exist" containerID="f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9"
Sep 17 23:16:19.723508 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.723466    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9"} err="failed to get container status \"f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9\": rpc error: code = NotFound desc = could not find container \"f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9\": container with ID starting with f5539867f62fed0f38c35a781a74f185823786888f469e45579cec245f61a8c9 not found: ID does not exist"
Sep 17 23:16:19.730859 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.730830    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86868d784b-ldsd9"]
Sep 17 23:16:19.734634 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:19.734607    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86868d784b-ldsd9"]
Sep 17 23:16:21.418712 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:21.418676    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="980d629d-f13e-4e65-bc9e-ee128658e7e5" path="/var/lib/kubelet/pods/980d629d-f13e-4e65-bc9e-ee128658e7e5/volumes"
Sep 17 23:16:32.583881 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.583802    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:16:32.584340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.584083    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="alertmanager" containerID="cri-o://a4f62269dd819c33039f6705ea90393931a1c220f01ab659d123af27c8757a39" gracePeriod=120
Sep 17 23:16:32.584340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.584201    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-metric" containerID="cri-o://fa169db794362201d834bb0ae9dd0136c6c6206c3e562b15f9f6e6ca093b9ade" gracePeriod=120
Sep 17 23:16:32.584340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.584202    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-web" containerID="cri-o://303608bd729c8a9ba917f8d62e5ee881b64d817b94f90fcd17bcf7e71c473bc0" gracePeriod=120
Sep 17 23:16:32.584340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.584251    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="config-reloader" containerID="cri-o://bae58a33fb6af10dc29a6f1afea63f49c9dafbbe0b2ad8b500006852432ee484" gracePeriod=120
Sep 17 23:16:32.584340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.584204    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="prom-label-proxy" containerID="cri-o://ae071b436a785956f609daca21876c83e78a5a4dd1251c4955d926613c00c54e" gracePeriod=120
Sep 17 23:16:32.584340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.584231    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy" containerID="cri-o://59b8fa231afbdd9206a65e2124d37fa5a874392b74ebd65c72f733da2145e0eb" gracePeriod=120
Sep 17 23:16:32.768556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768525    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerID="ae071b436a785956f609daca21876c83e78a5a4dd1251c4955d926613c00c54e" exitCode=0
Sep 17 23:16:32.768556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768551    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerID="59b8fa231afbdd9206a65e2124d37fa5a874392b74ebd65c72f733da2145e0eb" exitCode=0
Sep 17 23:16:32.768556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768558    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerID="bae58a33fb6af10dc29a6f1afea63f49c9dafbbe0b2ad8b500006852432ee484" exitCode=0
Sep 17 23:16:32.768556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768566    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerID="a4f62269dd819c33039f6705ea90393931a1c220f01ab659d123af27c8757a39" exitCode=0
Sep 17 23:16:32.768808 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"ae071b436a785956f609daca21876c83e78a5a4dd1251c4955d926613c00c54e"}
Sep 17 23:16:32.768808 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768652    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"59b8fa231afbdd9206a65e2124d37fa5a874392b74ebd65c72f733da2145e0eb"}
Sep 17 23:16:32.768808 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"bae58a33fb6af10dc29a6f1afea63f49c9dafbbe0b2ad8b500006852432ee484"}
Sep 17 23:16:32.768808 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:32.768704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"a4f62269dd819c33039f6705ea90393931a1c220f01ab659d123af27c8757a39"}
Sep 17 23:16:33.778654 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.778616    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerID="fa169db794362201d834bb0ae9dd0136c6c6206c3e562b15f9f6e6ca093b9ade" exitCode=0
Sep 17 23:16:33.778654 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.778651    2562 generic.go:356] "Generic (PLEG): container finished" podID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerID="303608bd729c8a9ba917f8d62e5ee881b64d817b94f90fcd17bcf7e71c473bc0" exitCode=0
Sep 17 23:16:33.779068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.778680    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"fa169db794362201d834bb0ae9dd0136c6c6206c3e562b15f9f6e6ca093b9ade"}
Sep 17 23:16:33.779068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.778772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"303608bd729c8a9ba917f8d62e5ee881b64d817b94f90fcd17bcf7e71c473bc0"}
Sep 17 23:16:33.828510 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.828487    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:33.844534 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844512    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-out\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.844684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844549    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.844684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844567    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-tls-assets\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.844684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844588    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-web-config\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.844684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844607    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n9lt\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-kube-api-access-9n9lt\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.844684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844638    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-main-db\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.844684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844656    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-web\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.844684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844678    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-metric\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.845061 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844711    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-trusted-ca-bundle\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.845061 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844744    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-main-tls\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.845061 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844812    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-cluster-tls-config\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.845061 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844850    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-volume\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.845061 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.844996    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:16:33.845300 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.845164    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-main-db\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.845300 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.845241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:33.847454 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.847421    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-out" (OuterVolumeSpecName: "config-out") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:16:33.847886 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.847860    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-volume" (OuterVolumeSpecName: "config-volume") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:33.848015 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.847929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:33.848015 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.847943    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:33.848015 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.847960    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-kube-api-access-9n9lt" (OuterVolumeSpecName: "kube-api-access-9n9lt") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "kube-api-access-9n9lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:33.848293 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.848258    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:33.848367 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.848339    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:16:33.849154 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.849126    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:33.852371 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.852316    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:33.858812 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.858784    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-web-config" (OuterVolumeSpecName: "web-config") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:16:33.945711 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945626    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-metrics-client-ca\") pod \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\" (UID: \"dc402b2e-6ba3-4f30-a24b-ac265bc74710\") "
Sep 17 23:16:33.945851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945786    2562 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-out\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.945851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945805    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.945851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945818    2562 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-tls-assets\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.945851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945831    2562 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-web-config\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.945851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945844    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n9lt\" (UniqueName: \"kubernetes.io/projected/dc402b2e-6ba3-4f30-a24b-ac265bc74710-kube-api-access-9n9lt\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.946032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945858    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.946032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945872    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.946032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945885    2562 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.946032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945919    2562 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-secret-alertmanager-main-tls\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.946032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945933    2562 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-cluster-tls-config\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.946032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.945946    2562 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/dc402b2e-6ba3-4f30-a24b-ac265bc74710-config-volume\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:33.946202 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:33.946038    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "dc402b2e-6ba3-4f30-a24b-ac265bc74710" (UID: "dc402b2e-6ba3-4f30-a24b-ac265bc74710"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:16:34.046768 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.046735    2562 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dc402b2e-6ba3-4f30-a24b-ac265bc74710-metrics-client-ca\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:16:34.785462 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.785433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"dc402b2e-6ba3-4f30-a24b-ac265bc74710","Type":"ContainerDied","Data":"eab1d19ed869e7899b39ad56c3a492c25b6b502f1f99c53a8232981ea4367b18"}
Sep 17 23:16:34.785874 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.785471    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.785874 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.785505    2562 scope.go:117] "RemoveContainer" containerID="ae071b436a785956f609daca21876c83e78a5a4dd1251c4955d926613c00c54e"
Sep 17 23:16:34.793676 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.793650    2562 scope.go:117] "RemoveContainer" containerID="fa169db794362201d834bb0ae9dd0136c6c6206c3e562b15f9f6e6ca093b9ade"
Sep 17 23:16:34.800748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.800733    2562 scope.go:117] "RemoveContainer" containerID="59b8fa231afbdd9206a65e2124d37fa5a874392b74ebd65c72f733da2145e0eb"
Sep 17 23:16:34.808507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.808484    2562 scope.go:117] "RemoveContainer" containerID="303608bd729c8a9ba917f8d62e5ee881b64d817b94f90fcd17bcf7e71c473bc0"
Sep 17 23:16:34.808705 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.808688    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:16:34.813556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.813531    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:16:34.815821 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.815799    2562 scope.go:117] "RemoveContainer" containerID="bae58a33fb6af10dc29a6f1afea63f49c9dafbbe0b2ad8b500006852432ee484"
Sep 17 23:16:34.822542 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.822527    2562 scope.go:117] "RemoveContainer" containerID="a4f62269dd819c33039f6705ea90393931a1c220f01ab659d123af27c8757a39"
Sep 17 23:16:34.829508 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.829491    2562 scope.go:117] "RemoveContainer" containerID="40017622c7139f905853017748e87f95281f001cbb376fa7b144f7771d51bd91"
Sep 17 23:16:34.842414 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842389    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:16:34.842710 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842694    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8" containerName="registry"
Sep 17 23:16:34.842781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842713    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8" containerName="registry"
Sep 17 23:16:34.842781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842725    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy"
Sep 17 23:16:34.842781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842733    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy"
Sep 17 23:16:34.842781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842746    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="980d629d-f13e-4e65-bc9e-ee128658e7e5" containerName="console"
Sep 17 23:16:34.842781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842755    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="980d629d-f13e-4e65-bc9e-ee128658e7e5" containerName="console"
Sep 17 23:16:34.842781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842767    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="config-reloader"
Sep 17 23:16:34.842781 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842776    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="config-reloader"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842784    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-web"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842793    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-web"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842802    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="prom-label-proxy"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842810    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="prom-label-proxy"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842828    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="init-config-reloader"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842837    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="init-config-reloader"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842851    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="alertmanager"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842860    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="alertmanager"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842873    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-metric"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842881    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-metric"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842977    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="alertmanager"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.842990    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-web"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.843000    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="config-reloader"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.843008    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy-metric"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.843017    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d957c71e-eb0d-4976-97d7-21058a4171c8" containerName="registry"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.843026    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="kube-rbac-proxy"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.843034    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" containerName="prom-label-proxy"
Sep 17 23:16:34.843121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.843043    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="980d629d-f13e-4e65-bc9e-ee128658e7e5" containerName="console"
Sep 17 23:16:34.846731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.846710    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.849074 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849056    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 23:16:34.849186 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849136    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 23:16:34.849186 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849138    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 23:16:34.849307 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849213    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 23:16:34.849506 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849491    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 23:16:34.849506 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849500    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 23:16:34.849622 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849564    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 23:16:34.849684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849668    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fzb2d\""
Sep 17 23:16:34.849757 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.849740    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 23:16:34.850642 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850625    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.850709 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.850709 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.850792 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-web-config\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.850850 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.850918 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850863    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.850957 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.851003 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850961    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-config-volume\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.851003 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.850982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.851081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.851018    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-config-out\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.851081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.851040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.851081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.851079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.851179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.851105    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6m2q\" (UniqueName: \"kubernetes.io/projected/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-kube-api-access-s6m2q\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.854214 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.854195    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 23:16:34.861701 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.861678    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:16:34.952321 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952226    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952321 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952263    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-config-volume\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952533 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952511    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952573 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952547    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-config-out\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952661 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952624    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952661 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952650    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6m2q\" (UniqueName: \"kubernetes.io/projected/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-kube-api-access-s6m2q\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952742 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952742 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952706    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952814 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952814 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952794    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-web-config\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952887 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952827    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.952887 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.952859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.954402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.953223    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.954402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.953711    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.954402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.954355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.955399 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.955373    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.955843 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.955822    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.955990 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.955972    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-config-volume\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.956063 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.955975    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.956370 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.956350    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-web-config\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.956488 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.956469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.956583 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.956566    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-config-out\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.956866 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.956851    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.957206 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.957188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:34.960245 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:34.960226    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6m2q\" (UniqueName: \"kubernetes.io/projected/51c1b7ad-b451-4bbf-bf86-00920f46fdf0-kube-api-access-s6m2q\") pod \"alertmanager-main-0\" (UID: \"51c1b7ad-b451-4bbf-bf86-00920f46fdf0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:35.157190 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:35.157150    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 23:16:35.418669 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:35.418630    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc402b2e-6ba3-4f30-a24b-ac265bc74710" path="/var/lib/kubelet/pods/dc402b2e-6ba3-4f30-a24b-ac265bc74710/volumes"
Sep 17 23:16:35.442494 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:35.442471    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 23:16:35.444263 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:16:35.444240    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c1b7ad_b451_4bbf_bf86_00920f46fdf0.slice/crio-b3f7ab9ed28e2c8b37d8bdbba8ccd1ffaa79c7032c46d26d32a04fceea9684a8 WatchSource:0}: Error finding container b3f7ab9ed28e2c8b37d8bdbba8ccd1ffaa79c7032c46d26d32a04fceea9684a8: Status 404 returned error can't find the container with id b3f7ab9ed28e2c8b37d8bdbba8ccd1ffaa79c7032c46d26d32a04fceea9684a8
Sep 17 23:16:35.791266 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:35.791171    2562 generic.go:356] "Generic (PLEG): container finished" podID="51c1b7ad-b451-4bbf-bf86-00920f46fdf0" containerID="3ca4d1dad754d0c5bbcc67ac22580a65344381035d2b2898babcf6779a18b2d3" exitCode=0
Sep 17 23:16:35.791266 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:35.791256    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerDied","Data":"3ca4d1dad754d0c5bbcc67ac22580a65344381035d2b2898babcf6779a18b2d3"}
Sep 17 23:16:35.791699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:35.791324    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerStarted","Data":"b3f7ab9ed28e2c8b37d8bdbba8ccd1ffaa79c7032c46d26d32a04fceea9684a8"}
Sep 17 23:16:36.800199 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:36.800153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerStarted","Data":"cc03114f630ddb8d42f1a05514065746c0bc178518e088bff2dcd04af0ef2e79"}
Sep 17 23:16:36.800641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:36.800240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerStarted","Data":"3978407efeec890fe668551cc9e72ee7b48f9436ad41ec65d0ef52febe2e1163"}
Sep 17 23:16:36.800641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:36.800282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerStarted","Data":"f9a39115e4b3291e436edbf81454b6081b370f7b90704ee35f56fc923c1cbb57"}
Sep 17 23:16:36.800641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:36.800323    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerStarted","Data":"14ea57c74253ec06d579b585a3412acd1d8d6b72b348758b929a7b9765c1cbdd"}
Sep 17 23:16:36.800641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:36.800358    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerStarted","Data":"5d86cb2d310ae57f873a443b182b4132ec84c62be63420aa03e412986c7dc778"}
Sep 17 23:16:36.800641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:36.800395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51c1b7ad-b451-4bbf-bf86-00920f46fdf0","Type":"ContainerStarted","Data":"1f793d70cc907e29305bac0abecc2d0c5be14b7a9df8563c385daa21a88e7ab7"}
Sep 17 23:16:36.832282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:36.832235    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.817634009 podStartE2EDuration="2.832220986s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.586977ms" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:36.830383242 +0000 UTC m=+357.924910282" watchObservedRunningTime="2026-09-17 23:16:36.832220986 +0000 UTC m=+357.926748027"
Sep 17 23:16:37.397823 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.397790    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cbd9bbc45-wsr7d"]
Sep 17 23:16:37.399924 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.399888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.402288 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.402269    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 23:16:37.402518 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.402491    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 23:16:37.402690 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.402529    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 23:16:37.402690 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.402568    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hjfdq\""
Sep 17 23:16:37.403681 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.403665    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 23:16:37.403765 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.403731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 23:16:37.407564 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.407544    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 23:16:37.419463 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.419437    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbd9bbc45-wsr7d"]
Sep 17 23:16:37.471660 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.471624    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-oauth-serving-cert\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.471831 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.471672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8tr\" (UniqueName: \"kubernetes.io/projected/b5b9eb09-84fc-4736-9039-8de7109640ea-kube-api-access-zk8tr\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.471831 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.471805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-console-config\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.471931 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.471852    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-serving-cert\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.471931 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.471872    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-trusted-ca-bundle\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.472000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.471939    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-service-ca\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.472000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.471992    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-oauth-config\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.572337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.572301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-oauth-config\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.572337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.572338    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-oauth-serving-cert\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.572533 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.572432    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8tr\" (UniqueName: \"kubernetes.io/projected/b5b9eb09-84fc-4736-9039-8de7109640ea-kube-api-access-zk8tr\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.572569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.572539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-console-config\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.572619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.572601    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-serving-cert\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.572662 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.572639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-trusted-ca-bundle\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.572704 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.572674    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-service-ca\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.573234 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.573213    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-console-config\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.573348 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.573252    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-oauth-serving-cert\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.573348 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.573325    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-service-ca\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.573836 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.573816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-trusted-ca-bundle\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.574941 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.574923    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-oauth-config\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.575071 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.575055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-serving-cert\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.580381 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.580363    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8tr\" (UniqueName: \"kubernetes.io/projected/b5b9eb09-84fc-4736-9039-8de7109640ea-kube-api-access-zk8tr\") pod \"console-cbd9bbc45-wsr7d\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") " pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.709840 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.709740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:37.979713 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:37.979690    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cbd9bbc45-wsr7d"]
Sep 17 23:16:37.982028 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:16:37.982004    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b9eb09_84fc_4736_9039_8de7109640ea.slice/crio-108c370ec669a70ed2659653027c4faca2702817159b2c9b26989c6bd38b513f WatchSource:0}: Error finding container 108c370ec669a70ed2659653027c4faca2702817159b2c9b26989c6bd38b513f: Status 404 returned error can't find the container with id 108c370ec669a70ed2659653027c4faca2702817159b2c9b26989c6bd38b513f
Sep 17 23:16:38.810767 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:38.810730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd9bbc45-wsr7d" event={"ID":"b5b9eb09-84fc-4736-9039-8de7109640ea","Type":"ContainerStarted","Data":"9d0edee7f7f999e76f33a2f617503f6b470f7ea4e80b1e5d3617475a73742a9f"}
Sep 17 23:16:38.810970 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:38.810784    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd9bbc45-wsr7d" event={"ID":"b5b9eb09-84fc-4736-9039-8de7109640ea","Type":"ContainerStarted","Data":"108c370ec669a70ed2659653027c4faca2702817159b2c9b26989c6bd38b513f"}
Sep 17 23:16:38.827524 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:38.827411    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cbd9bbc45-wsr7d" podStartSLOduration=1.827394243 podStartE2EDuration="1.827394243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:16:38.82708957 +0000 UTC m=+359.921616605" watchObservedRunningTime="2026-09-17 23:16:38.827394243 +0000 UTC m=+359.921921280"
Sep 17 23:16:47.710177 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:47.710136    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:47.710649 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:47.710207    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:47.715197 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:47.715172    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:16:47.851496 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:16:47.851467    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:17:37.737266 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.737229    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"]
Sep 17 23:17:37.739531 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.739514    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.742224 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.742202    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 23:17:37.742366 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.742249    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 23:17:37.742366 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.742320    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vnhk2\""
Sep 17 23:17:37.748524 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.748500    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"]
Sep 17 23:17:37.856380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.856350    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.856380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.856386    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.856585 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.856430    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8x7v\" (UniqueName: \"kubernetes.io/projected/a1562dc8-5f44-4762-9711-04f2a21d572b-kube-api-access-p8x7v\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.957696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.957662    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8x7v\" (UniqueName: \"kubernetes.io/projected/a1562dc8-5f44-4762-9711-04f2a21d572b-kube-api-access-p8x7v\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.957761 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.957730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.957761 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.957754    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.958114 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.958097    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.958169 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.958128    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:37.966609 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:37.966581    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8x7v\" (UniqueName: \"kubernetes.io/projected/a1562dc8-5f44-4762-9711-04f2a21d572b-kube-api-access-p8x7v\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:38.049465 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:38.049395    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:38.317631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:38.317552    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"]
Sep 17 23:17:38.320505 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:17:38.320475    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1562dc8_5f44_4762_9711_04f2a21d572b.slice/crio-b4796a61eaae41b7be43035715a108344e46a7c7b2205bd300abfbe3bc882eda WatchSource:0}: Error finding container b4796a61eaae41b7be43035715a108344e46a7c7b2205bd300abfbe3bc882eda: Status 404 returned error can't find the container with id b4796a61eaae41b7be43035715a108344e46a7c7b2205bd300abfbe3bc882eda
Sep 17 23:17:39.054501 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:39.054464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4" event={"ID":"a1562dc8-5f44-4762-9711-04f2a21d572b","Type":"ContainerStarted","Data":"b4796a61eaae41b7be43035715a108344e46a7c7b2205bd300abfbe3bc882eda"}
Sep 17 23:17:45.082823 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:45.082783    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerID="f78af6b7da987c7fe3854f9000a4cd667510317500af26baad1a5095e4e1426d" exitCode=0
Sep 17 23:17:45.083242 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:45.082872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4" event={"ID":"a1562dc8-5f44-4762-9711-04f2a21d572b","Type":"ContainerDied","Data":"f78af6b7da987c7fe3854f9000a4cd667510317500af26baad1a5095e4e1426d"}
Sep 17 23:17:47.093553 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:47.093521    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerID="214829f796152616f4c9fa4349e0df188c088c10288f12093f105a42f413e448" exitCode=0
Sep 17 23:17:47.093916 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:47.093581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4" event={"ID":"a1562dc8-5f44-4762-9711-04f2a21d572b","Type":"ContainerDied","Data":"214829f796152616f4c9fa4349e0df188c088c10288f12093f105a42f413e448"}
Sep 17 23:17:54.125798 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:54.125760    2562 generic.go:356] "Generic (PLEG): container finished" podID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerID="589cce0796c7ff7d4cac822f26dcaec5b8366141e23b10829b9382aed9717fd3" exitCode=0
Sep 17 23:17:54.126221 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:54.125838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4" event={"ID":"a1562dc8-5f44-4762-9711-04f2a21d572b","Type":"ContainerDied","Data":"589cce0796c7ff7d4cac822f26dcaec5b8366141e23b10829b9382aed9717fd3"}
Sep 17 23:17:55.253211 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.253187    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:55.290550 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.290518    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8x7v\" (UniqueName: \"kubernetes.io/projected/a1562dc8-5f44-4762-9711-04f2a21d572b-kube-api-access-p8x7v\") pod \"a1562dc8-5f44-4762-9711-04f2a21d572b\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") "
Sep 17 23:17:55.290712 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.290569    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-util\") pod \"a1562dc8-5f44-4762-9711-04f2a21d572b\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") "
Sep 17 23:17:55.290712 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.290626    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-bundle\") pod \"a1562dc8-5f44-4762-9711-04f2a21d572b\" (UID: \"a1562dc8-5f44-4762-9711-04f2a21d572b\") "
Sep 17 23:17:55.291297 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.291270    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-bundle" (OuterVolumeSpecName: "bundle") pod "a1562dc8-5f44-4762-9711-04f2a21d572b" (UID: "a1562dc8-5f44-4762-9711-04f2a21d572b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:17:55.292720 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.292691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1562dc8-5f44-4762-9711-04f2a21d572b-kube-api-access-p8x7v" (OuterVolumeSpecName: "kube-api-access-p8x7v") pod "a1562dc8-5f44-4762-9711-04f2a21d572b" (UID: "a1562dc8-5f44-4762-9711-04f2a21d572b"). InnerVolumeSpecName "kube-api-access-p8x7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:17:55.295097 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.295074    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-util" (OuterVolumeSpecName: "util") pod "a1562dc8-5f44-4762-9711-04f2a21d572b" (UID: "a1562dc8-5f44-4762-9711-04f2a21d572b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:17:55.391647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.391560    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-util\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:17:55.391647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.391589    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a1562dc8-5f44-4762-9711-04f2a21d572b-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:17:55.391647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:55.391597    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8x7v\" (UniqueName: \"kubernetes.io/projected/a1562dc8-5f44-4762-9711-04f2a21d572b-kube-api-access-p8x7v\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:17:56.135990 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:56.135959    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4"
Sep 17 23:17:56.135990 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:56.135969    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1alt4j4" event={"ID":"a1562dc8-5f44-4762-9711-04f2a21d572b","Type":"ContainerDied","Data":"b4796a61eaae41b7be43035715a108344e46a7c7b2205bd300abfbe3bc882eda"}
Sep 17 23:17:56.136245 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:56.136013    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4796a61eaae41b7be43035715a108344e46a7c7b2205bd300abfbe3bc882eda"
Sep 17 23:17:59.906822 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.906788    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"]
Sep 17 23:17:59.907229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.907113    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerName="pull"
Sep 17 23:17:59.907229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.907124    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerName="pull"
Sep 17 23:17:59.907229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.907133    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerName="util"
Sep 17 23:17:59.907229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.907140    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerName="util"
Sep 17 23:17:59.907229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.907146    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerName="extract"
Sep 17 23:17:59.907229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.907152    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerName="extract"
Sep 17 23:17:59.907455 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.907241    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1562dc8-5f44-4762-9711-04f2a21d572b" containerName="extract"
Sep 17 23:17:59.910365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.910349    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:17:59.913066 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.913044    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 23:17:59.913066 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.913064    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 23:17:59.913279 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.913169    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 23:17:59.913339 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.913316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-z2hmx\""
Sep 17 23:17:59.921508 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.921484    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"]
Sep 17 23:17:59.927070 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.927050    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e27fb77-6cb8-4a4f-8dfe-935fed8cc728-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z6r8t\" (UID: \"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:17:59.927149 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:17:59.927102    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqxn\" (UniqueName: \"kubernetes.io/projected/5e27fb77-6cb8-4a4f-8dfe-935fed8cc728-kube-api-access-cgqxn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z6r8t\" (UID: \"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:00.028130 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:00.028094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqxn\" (UniqueName: \"kubernetes.io/projected/5e27fb77-6cb8-4a4f-8dfe-935fed8cc728-kube-api-access-cgqxn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z6r8t\" (UID: \"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:00.028289 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:00.028156    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e27fb77-6cb8-4a4f-8dfe-935fed8cc728-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z6r8t\" (UID: \"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:00.030597 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:00.030576    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/5e27fb77-6cb8-4a4f-8dfe-935fed8cc728-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z6r8t\" (UID: \"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:00.037354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:00.037330    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqxn\" (UniqueName: \"kubernetes.io/projected/5e27fb77-6cb8-4a4f-8dfe-935fed8cc728-kube-api-access-cgqxn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-z6r8t\" (UID: \"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:00.220511 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:00.220421    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:00.494970 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:00.494942    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"]
Sep 17 23:18:00.496594 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:18:00.496567    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e27fb77_6cb8_4a4f_8dfe_935fed8cc728.slice/crio-5b5f1c0f23e9138c4ac28372c52ab57f6a5ef2ab86c69c29fee00e42790b174c WatchSource:0}: Error finding container 5b5f1c0f23e9138c4ac28372c52ab57f6a5ef2ab86c69c29fee00e42790b174c: Status 404 returned error can't find the container with id 5b5f1c0f23e9138c4ac28372c52ab57f6a5ef2ab86c69c29fee00e42790b174c
Sep 17 23:18:01.158803 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:01.158768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t" event={"ID":"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728","Type":"ContainerStarted","Data":"5b5f1c0f23e9138c4ac28372c52ab57f6a5ef2ab86c69c29fee00e42790b174c"}
Sep 17 23:18:03.825132 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.825097    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-q8xvp"]
Sep 17 23:18:03.828582 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.828565    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.831425 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.831398    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 23:18:03.831545 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.831444    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 23:18:03.831545 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.831482    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tnmsm\""
Sep 17 23:18:03.837279 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.837252    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-q8xvp"]
Sep 17 23:18:03.861312 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.861276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp2fc\" (UniqueName: \"kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-kube-api-access-dp2fc\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.861494 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.861324    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-certificates\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.861494 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.861339    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-cabundle0\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.961806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.961770    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-certificates\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.961806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.961804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-cabundle0\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.962096 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.961865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp2fc\" (UniqueName: \"kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-kube-api-access-dp2fc\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.962096 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:18:03.961970    2562 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 17 23:18:03.962096 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:18:03.961995    2562 secret.go:281] references non-existent secret key: ca.crt
Sep 17 23:18:03.962096 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:18:03.962006    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 23:18:03.962096 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:18:03.962021    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-q8xvp: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 23:18:03.962096 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:18:03.962099    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-certificates podName:eafa9ef9-a7d8-4cc3-9223-f026b8e61265 nodeName:}" failed. No retries permitted until 2026-09-17 23:18:04.462077348 +0000 UTC m=+445.556604382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-certificates") pod "keda-operator-579c8958f7-q8xvp" (UID: "eafa9ef9-a7d8-4cc3-9223-f026b8e61265") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 17 23:18:03.962663 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.962642    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-cabundle0\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:03.972768 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:03.972745    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp2fc\" (UniqueName: \"kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-kube-api-access-dp2fc\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:04.174167 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.174113    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t" event={"ID":"5e27fb77-6cb8-4a4f-8dfe-935fed8cc728","Type":"ContainerStarted","Data":"3fa942194ba1bab985e39c30fa523ebdd24fca38b72849fa3f35b8750b94461b"}
Sep 17 23:18:04.174345 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.174294    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:04.195757 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.195703    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t" podStartSLOduration=2.571870137 podStartE2EDuration="5.195684321s" totalImagesPullingTime="2.623814184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:04.1936506 +0000 UTC m=+445.288177636" watchObservedRunningTime="2026-09-17 23:18:04.195684321 +0000 UTC m=+445.290211356"
Sep 17 23:18:04.243554 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.243520    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9hth2"]
Sep 17 23:18:04.247075 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.247055    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.249578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.249558    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 23:18:04.255330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.255077    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9hth2"]
Sep 17 23:18:04.365347 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.365305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2da494e0-87f1-4400-ae6e-4b7aeddcee0d-certificates\") pod \"keda-admission-6c4d864b86-9hth2\" (UID: \"2da494e0-87f1-4400-ae6e-4b7aeddcee0d\") " pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.365347 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.365345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-897f8\" (UniqueName: \"kubernetes.io/projected/2da494e0-87f1-4400-ae6e-4b7aeddcee0d-kube-api-access-897f8\") pod \"keda-admission-6c4d864b86-9hth2\" (UID: \"2da494e0-87f1-4400-ae6e-4b7aeddcee0d\") " pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.466317 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.466221    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-certificates\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:04.466317 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.466282    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2da494e0-87f1-4400-ae6e-4b7aeddcee0d-certificates\") pod \"keda-admission-6c4d864b86-9hth2\" (UID: \"2da494e0-87f1-4400-ae6e-4b7aeddcee0d\") " pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.466317 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.466298    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-897f8\" (UniqueName: \"kubernetes.io/projected/2da494e0-87f1-4400-ae6e-4b7aeddcee0d-kube-api-access-897f8\") pod \"keda-admission-6c4d864b86-9hth2\" (UID: \"2da494e0-87f1-4400-ae6e-4b7aeddcee0d\") " pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.468818 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.468783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/eafa9ef9-a7d8-4cc3-9223-f026b8e61265-certificates\") pod \"keda-operator-579c8958f7-q8xvp\" (UID: \"eafa9ef9-a7d8-4cc3-9223-f026b8e61265\") " pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:04.468818 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.468795    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/2da494e0-87f1-4400-ae6e-4b7aeddcee0d-certificates\") pod \"keda-admission-6c4d864b86-9hth2\" (UID: \"2da494e0-87f1-4400-ae6e-4b7aeddcee0d\") " pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.475310 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.475276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-897f8\" (UniqueName: \"kubernetes.io/projected/2da494e0-87f1-4400-ae6e-4b7aeddcee0d-kube-api-access-897f8\") pod \"keda-admission-6c4d864b86-9hth2\" (UID: \"2da494e0-87f1-4400-ae6e-4b7aeddcee0d\") " pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.559600 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.559566    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:04.739776 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.739690    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:04.830030 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:04.830005    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-9hth2"]
Sep 17 23:18:04.831910 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:18:04.831872    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da494e0_87f1_4400_ae6e_4b7aeddcee0d.slice/crio-5c3051cde44f38525f65f60ddcc1cee53eb64cf0660e956f9da6e5818dbd92db WatchSource:0}: Error finding container 5c3051cde44f38525f65f60ddcc1cee53eb64cf0660e956f9da6e5818dbd92db: Status 404 returned error can't find the container with id 5c3051cde44f38525f65f60ddcc1cee53eb64cf0660e956f9da6e5818dbd92db
Sep 17 23:18:05.011431 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:05.011399    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-q8xvp"]
Sep 17 23:18:05.014388 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:18:05.014361    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafa9ef9_a7d8_4cc3_9223_f026b8e61265.slice/crio-207db0de06e46c9e071ab2e62bf0a56d175d98950458ea74f80527bd0fe8df05 WatchSource:0}: Error finding container 207db0de06e46c9e071ab2e62bf0a56d175d98950458ea74f80527bd0fe8df05: Status 404 returned error can't find the container with id 207db0de06e46c9e071ab2e62bf0a56d175d98950458ea74f80527bd0fe8df05
Sep 17 23:18:05.180237 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:05.180192    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-q8xvp" event={"ID":"eafa9ef9-a7d8-4cc3-9223-f026b8e61265","Type":"ContainerStarted","Data":"207db0de06e46c9e071ab2e62bf0a56d175d98950458ea74f80527bd0fe8df05"}
Sep 17 23:18:05.181269 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:05.181242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9hth2" event={"ID":"2da494e0-87f1-4400-ae6e-4b7aeddcee0d","Type":"ContainerStarted","Data":"5c3051cde44f38525f65f60ddcc1cee53eb64cf0660e956f9da6e5818dbd92db"}
Sep 17 23:18:07.194506 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:07.194462    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-9hth2" event={"ID":"2da494e0-87f1-4400-ae6e-4b7aeddcee0d","Type":"ContainerStarted","Data":"94ea81e5e69e294940c8dec1a952d93406548660d48599ebad20149a243f78c6"}
Sep 17 23:18:07.195010 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:07.194586    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:07.212979 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:07.212933    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-9hth2" podStartSLOduration=1.5376748660000001 podStartE2EDuration="3.212911065s" totalImagesPullingTime="1.675236199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:07.211917172 +0000 UTC m=+448.306444200" watchObservedRunningTime="2026-09-17 23:18:07.212911065 +0000 UTC m=+448.307438097"
Sep 17 23:18:08.200511 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:08.200470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-q8xvp" event={"ID":"eafa9ef9-a7d8-4cc3-9223-f026b8e61265","Type":"ContainerStarted","Data":"98cebb1b3d7b6f2f3d3166dad056cc7055f4b805a80df70cdc286d145cc398a5"}
Sep 17 23:18:08.200912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:08.200566    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:08.219157 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:08.219105    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-q8xvp" podStartSLOduration=2.116456669 podStartE2EDuration="5.219089651s" totalImagesPullingTime="3.102632982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:18:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:18:08.216785214 +0000 UTC m=+449.311312250" watchObservedRunningTime="2026-09-17 23:18:08.219089651 +0000 UTC m=+449.313616729"
Sep 17 23:18:25.184333 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:25.184301    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-z6r8t"
Sep 17 23:18:28.203600 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:28.203563    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-9hth2"
Sep 17 23:18:29.207417 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:29.207388    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-q8xvp"
Sep 17 23:18:57.623919 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.623872    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"]
Sep 17 23:18:57.628851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.628831    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.631503 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.631480    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vnhk2\""
Sep 17 23:18:57.631503 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.631502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 23:18:57.631665 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.631525    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 23:18:57.636635 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.636459    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"]
Sep 17 23:18:57.706434 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.706395    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kxnk\" (UniqueName: \"kubernetes.io/projected/83f8d730-69e6-407b-bfef-8f0d813356a6-kube-api-access-7kxnk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.706626 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.706471    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.706626 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.706533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.806979 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.806935    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.807157 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.807000    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.807157 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.807120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kxnk\" (UniqueName: \"kubernetes.io/projected/83f8d730-69e6-407b-bfef-8f0d813356a6-kube-api-access-7kxnk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.807362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.807347    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.807403 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.807365    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.815919 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.815862    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kxnk\" (UniqueName: \"kubernetes.io/projected/83f8d730-69e6-407b-bfef-8f0d813356a6-kube-api-access-7kxnk\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:57.938701 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:57.938595    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:18:58.207696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:58.207670    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"]
Sep 17 23:18:58.209785 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:18:58.209753    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f8d730_69e6_407b_bfef_8f0d813356a6.slice/crio-c43691787ee27df1fa28f14722fd1cc91b10dbc13820b925938910e1799f9b31 WatchSource:0}: Error finding container c43691787ee27df1fa28f14722fd1cc91b10dbc13820b925938910e1799f9b31: Status 404 returned error can't find the container with id c43691787ee27df1fa28f14722fd1cc91b10dbc13820b925938910e1799f9b31
Sep 17 23:18:58.428486 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:58.428449    2562 generic.go:356] "Generic (PLEG): container finished" podID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerID="d3dcfbc082292f34048fb543145fa3bffc7025d8c254ee9dee82f98fff64ddbe" exitCode=0
Sep 17 23:18:58.428486 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:58.428488    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq" event={"ID":"83f8d730-69e6-407b-bfef-8f0d813356a6","Type":"ContainerDied","Data":"d3dcfbc082292f34048fb543145fa3bffc7025d8c254ee9dee82f98fff64ddbe"}
Sep 17 23:18:58.428738 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:18:58.428523    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq" event={"ID":"83f8d730-69e6-407b-bfef-8f0d813356a6","Type":"ContainerStarted","Data":"c43691787ee27df1fa28f14722fd1cc91b10dbc13820b925938910e1799f9b31"}
Sep 17 23:19:00.440244 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:00.440199    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq" event={"ID":"83f8d730-69e6-407b-bfef-8f0d813356a6","Type":"ContainerStarted","Data":"2ed1eb8daf2b2cc19bdd0ea9217719ef6a1841ed49f036977870062e5ee65e0e"}
Sep 17 23:19:01.446037 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:01.446003    2562 generic.go:356] "Generic (PLEG): container finished" podID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerID="2ed1eb8daf2b2cc19bdd0ea9217719ef6a1841ed49f036977870062e5ee65e0e" exitCode=0
Sep 17 23:19:01.446502 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:01.446052    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq" event={"ID":"83f8d730-69e6-407b-bfef-8f0d813356a6","Type":"ContainerDied","Data":"2ed1eb8daf2b2cc19bdd0ea9217719ef6a1841ed49f036977870062e5ee65e0e"}
Sep 17 23:19:02.453197 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:02.453160    2562 generic.go:356] "Generic (PLEG): container finished" podID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerID="2c5a1c36af25768d7bf1187734af473d1050aef888e21093610814761ed9e3b4" exitCode=0
Sep 17 23:19:02.453554 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:02.453240    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq" event={"ID":"83f8d730-69e6-407b-bfef-8f0d813356a6","Type":"ContainerDied","Data":"2c5a1c36af25768d7bf1187734af473d1050aef888e21093610814761ed9e3b4"}
Sep 17 23:19:03.576319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.576290    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:19:03.659667 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.659627    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-bundle\") pod \"83f8d730-69e6-407b-bfef-8f0d813356a6\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") "
Sep 17 23:19:03.659667 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.659675    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-util\") pod \"83f8d730-69e6-407b-bfef-8f0d813356a6\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") "
Sep 17 23:19:03.659891 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.659725    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kxnk\" (UniqueName: \"kubernetes.io/projected/83f8d730-69e6-407b-bfef-8f0d813356a6-kube-api-access-7kxnk\") pod \"83f8d730-69e6-407b-bfef-8f0d813356a6\" (UID: \"83f8d730-69e6-407b-bfef-8f0d813356a6\") "
Sep 17 23:19:03.660393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.660365    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-bundle" (OuterVolumeSpecName: "bundle") pod "83f8d730-69e6-407b-bfef-8f0d813356a6" (UID: "83f8d730-69e6-407b-bfef-8f0d813356a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:19:03.661825 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.661800    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f8d730-69e6-407b-bfef-8f0d813356a6-kube-api-access-7kxnk" (OuterVolumeSpecName: "kube-api-access-7kxnk") pod "83f8d730-69e6-407b-bfef-8f0d813356a6" (UID: "83f8d730-69e6-407b-bfef-8f0d813356a6"). InnerVolumeSpecName "kube-api-access-7kxnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:03.748418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.748325    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-util" (OuterVolumeSpecName: "util") pod "83f8d730-69e6-407b-bfef-8f0d813356a6" (UID: "83f8d730-69e6-407b-bfef-8f0d813356a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:19:03.761171 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.761144    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:19:03.761171 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.761169    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/83f8d730-69e6-407b-bfef-8f0d813356a6-util\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:19:03.761297 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:03.761180    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kxnk\" (UniqueName: \"kubernetes.io/projected/83f8d730-69e6-407b-bfef-8f0d813356a6-kube-api-access-7kxnk\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:19:04.463626 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:04.463598    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq"
Sep 17 23:19:04.463823 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:04.463632    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pv8fq" event={"ID":"83f8d730-69e6-407b-bfef-8f0d813356a6","Type":"ContainerDied","Data":"c43691787ee27df1fa28f14722fd1cc91b10dbc13820b925938910e1799f9b31"}
Sep 17 23:19:04.463823 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:04.463678    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c43691787ee27df1fa28f14722fd1cc91b10dbc13820b925938910e1799f9b31"
Sep 17 23:19:21.707549 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707518    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m8p7n"]
Sep 17 23:19:21.707944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707865    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerName="extract"
Sep 17 23:19:21.707944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707876    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerName="extract"
Sep 17 23:19:21.707944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707885    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerName="util"
Sep 17 23:19:21.707944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707890    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerName="util"
Sep 17 23:19:21.707944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707912    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerName="pull"
Sep 17 23:19:21.707944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707921    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerName="pull"
Sep 17 23:19:21.708132 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.707971    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f8d730-69e6-407b-bfef-8f0d813356a6" containerName="extract"
Sep 17 23:19:21.714929 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.714888    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:21.717791 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.717766    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 23:19:21.718575 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.718550    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 23:19:21.718693 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.718660    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6f6n4\""
Sep 17 23:19:21.719631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.719608    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m8p7n"]
Sep 17 23:19:21.814953 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.814922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t6vk\" (UniqueName: \"kubernetes.io/projected/cee1abbc-630d-44bf-a933-72246e14180c-kube-api-access-7t6vk\") pod \"cert-manager-webhook-6486ffd997-m8p7n\" (UID: \"cee1abbc-630d-44bf-a933-72246e14180c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:21.815133 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.814991    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cee1abbc-630d-44bf-a933-72246e14180c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m8p7n\" (UID: \"cee1abbc-630d-44bf-a933-72246e14180c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:21.915788 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.915756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t6vk\" (UniqueName: \"kubernetes.io/projected/cee1abbc-630d-44bf-a933-72246e14180c-kube-api-access-7t6vk\") pod \"cert-manager-webhook-6486ffd997-m8p7n\" (UID: \"cee1abbc-630d-44bf-a933-72246e14180c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:21.915974 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.915802    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cee1abbc-630d-44bf-a933-72246e14180c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m8p7n\" (UID: \"cee1abbc-630d-44bf-a933-72246e14180c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:21.925493 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.925468    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cee1abbc-630d-44bf-a933-72246e14180c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-m8p7n\" (UID: \"cee1abbc-630d-44bf-a933-72246e14180c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:21.925593 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:21.925565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t6vk\" (UniqueName: \"kubernetes.io/projected/cee1abbc-630d-44bf-a933-72246e14180c-kube-api-access-7t6vk\") pod \"cert-manager-webhook-6486ffd997-m8p7n\" (UID: \"cee1abbc-630d-44bf-a933-72246e14180c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:22.035860 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:22.035779    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:22.310918 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:22.308341    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-m8p7n"]
Sep 17 23:19:22.311631 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:19:22.311595    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcee1abbc_630d_44bf_a933_72246e14180c.slice/crio-6b2c78dd825775eb9b183cf1f160bfc1454681a8e8e473be4697a2d8585ddefc WatchSource:0}: Error finding container 6b2c78dd825775eb9b183cf1f160bfc1454681a8e8e473be4697a2d8585ddefc: Status 404 returned error can't find the container with id 6b2c78dd825775eb9b183cf1f160bfc1454681a8e8e473be4697a2d8585ddefc
Sep 17 23:19:22.547158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:22.547123    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n" event={"ID":"cee1abbc-630d-44bf-a933-72246e14180c","Type":"ContainerStarted","Data":"6b2c78dd825775eb9b183cf1f160bfc1454681a8e8e473be4697a2d8585ddefc"}
Sep 17 23:19:25.563462 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:25.563424    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n" event={"ID":"cee1abbc-630d-44bf-a933-72246e14180c","Type":"ContainerStarted","Data":"2f09c0b2021050136cb42712b15b84766c463e455ca23597a1b390219f871dfc"}
Sep 17 23:19:25.563834 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:25.563554    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:25.580191 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:25.580136    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n" podStartSLOduration=2.101714716 podStartE2EDuration="4.580119234s" totalImagesPullingTime="2.478404518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:25.579424379 +0000 UTC m=+526.673951413" watchObservedRunningTime="2026-09-17 23:19:25.580119234 +0000 UTC m=+526.674646269"
Sep 17 23:19:29.327984 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.327953    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"]
Sep 17 23:19:29.331505 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.331490    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.334197 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.334167    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vnhk2\""
Sep 17 23:19:29.334308 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.334170    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 23:19:29.334308 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.334227    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 23:19:29.340038 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.340018    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"]
Sep 17 23:19:29.485283 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.485240    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.485459 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.485290    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzj22\" (UniqueName: \"kubernetes.io/projected/0789ac23-3d31-402d-92a0-fc13fd562d1b-kube-api-access-hzj22\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.485459 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.485366    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.586151 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.586114    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.586297 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.586216    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.586297 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.586248    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzj22\" (UniqueName: \"kubernetes.io/projected/0789ac23-3d31-402d-92a0-fc13fd562d1b-kube-api-access-hzj22\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.586566 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.586546    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.586607 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.586562    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.593994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.593961    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzj22\" (UniqueName: \"kubernetes.io/projected/0789ac23-3d31-402d-92a0-fc13fd562d1b-kube-api-access-hzj22\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.640996 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.640957    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:29.912467 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:29.912371    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"]
Sep 17 23:19:29.917761 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:19:29.915186    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0789ac23_3d31_402d_92a0_fc13fd562d1b.slice/crio-1fefbed298b7eba6647b53ce0844f77149383537d5dc8e62ee79108cf95fd30f WatchSource:0}: Error finding container 1fefbed298b7eba6647b53ce0844f77149383537d5dc8e62ee79108cf95fd30f: Status 404 returned error can't find the container with id 1fefbed298b7eba6647b53ce0844f77149383537d5dc8e62ee79108cf95fd30f
Sep 17 23:19:30.586393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:30.586354    2562 generic.go:356] "Generic (PLEG): container finished" podID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerID="ba8a774476739e4a037b7e07b199cdea195ccc11c00cce170bf93227e8a0cf7e" exitCode=0
Sep 17 23:19:30.586930 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:30.586411    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj" event={"ID":"0789ac23-3d31-402d-92a0-fc13fd562d1b","Type":"ContainerDied","Data":"ba8a774476739e4a037b7e07b199cdea195ccc11c00cce170bf93227e8a0cf7e"}
Sep 17 23:19:30.586930 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:30.586460    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj" event={"ID":"0789ac23-3d31-402d-92a0-fc13fd562d1b","Type":"ContainerStarted","Data":"1fefbed298b7eba6647b53ce0844f77149383537d5dc8e62ee79108cf95fd30f"}
Sep 17 23:19:31.570875 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:31.570847    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-m8p7n"
Sep 17 23:19:33.602497 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:33.602461    2562 generic.go:356] "Generic (PLEG): container finished" podID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerID="2379f3cbf2433bca2ca32a93cfc02aea813bef4f07358808fdf9f39682c46499" exitCode=0
Sep 17 23:19:33.602872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:33.602503    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj" event={"ID":"0789ac23-3d31-402d-92a0-fc13fd562d1b","Type":"ContainerDied","Data":"2379f3cbf2433bca2ca32a93cfc02aea813bef4f07358808fdf9f39682c46499"}
Sep 17 23:19:34.609693 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:34.609656    2562 generic.go:356] "Generic (PLEG): container finished" podID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerID="db9741fefc0eea4dcb18d524df5143d83369d6d84cfd3740620ff795ab7f4175" exitCode=0
Sep 17 23:19:34.610136 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:34.609747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj" event={"ID":"0789ac23-3d31-402d-92a0-fc13fd562d1b","Type":"ContainerDied","Data":"db9741fefc0eea4dcb18d524df5143d83369d6d84cfd3740620ff795ab7f4175"}
Sep 17 23:19:35.738079 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.738053    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:35.842230 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.842198    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-bundle\") pod \"0789ac23-3d31-402d-92a0-fc13fd562d1b\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") "
Sep 17 23:19:35.842404 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.842266    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzj22\" (UniqueName: \"kubernetes.io/projected/0789ac23-3d31-402d-92a0-fc13fd562d1b-kube-api-access-hzj22\") pod \"0789ac23-3d31-402d-92a0-fc13fd562d1b\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") "
Sep 17 23:19:35.842404 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.842317    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-util\") pod \"0789ac23-3d31-402d-92a0-fc13fd562d1b\" (UID: \"0789ac23-3d31-402d-92a0-fc13fd562d1b\") "
Sep 17 23:19:35.842678 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.842654    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-bundle" (OuterVolumeSpecName: "bundle") pod "0789ac23-3d31-402d-92a0-fc13fd562d1b" (UID: "0789ac23-3d31-402d-92a0-fc13fd562d1b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:19:35.844345 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.844328    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0789ac23-3d31-402d-92a0-fc13fd562d1b-kube-api-access-hzj22" (OuterVolumeSpecName: "kube-api-access-hzj22") pod "0789ac23-3d31-402d-92a0-fc13fd562d1b" (UID: "0789ac23-3d31-402d-92a0-fc13fd562d1b"). InnerVolumeSpecName "kube-api-access-hzj22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:19:35.846877 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.846851    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-util" (OuterVolumeSpecName: "util") pod "0789ac23-3d31-402d-92a0-fc13fd562d1b" (UID: "0789ac23-3d31-402d-92a0-fc13fd562d1b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:19:35.943957 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.943879    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:19:35.943957 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.943918    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzj22\" (UniqueName: \"kubernetes.io/projected/0789ac23-3d31-402d-92a0-fc13fd562d1b-kube-api-access-hzj22\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:19:35.943957 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:35.943930    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0789ac23-3d31-402d-92a0-fc13fd562d1b-util\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:19:36.620965 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:36.620932    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj" event={"ID":"0789ac23-3d31-402d-92a0-fc13fd562d1b","Type":"ContainerDied","Data":"1fefbed298b7eba6647b53ce0844f77149383537d5dc8e62ee79108cf95fd30f"}
Sep 17 23:19:36.621138 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:36.620976    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fl7xzj"
Sep 17 23:19:36.621138 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:36.620981    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fefbed298b7eba6647b53ce0844f77149383537d5dc8e62ee79108cf95fd30f"
Sep 17 23:19:56.727485 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727455    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"]
Sep 17 23:19:56.727839 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727766    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerName="pull"
Sep 17 23:19:56.727839 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727776    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerName="pull"
Sep 17 23:19:56.727839 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727789    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerName="extract"
Sep 17 23:19:56.727839 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727794    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerName="extract"
Sep 17 23:19:56.727839 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727814    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerName="util"
Sep 17 23:19:56.727839 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727820    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerName="util"
Sep 17 23:19:56.728196 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.727869    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0789ac23-3d31-402d-92a0-fc13fd562d1b" containerName="extract"
Sep 17 23:19:56.736258 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.736234    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.739068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.739040    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"]
Sep 17 23:19:56.740553 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.740531    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 23:19:56.740717 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.740535    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 23:19:56.740878 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.740538    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 23:19:56.740990 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.740551    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cww64\""
Sep 17 23:19:56.741055 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.740566    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 23:19:56.741112 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.740662    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 23:19:56.815191 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.815158    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95538e94-8be7-4862-8873-75294372c415-metrics-cert\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.815362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.815200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95538e94-8be7-4862-8873-75294372c415-cert\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.815362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.815305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95538e94-8be7-4862-8873-75294372c415-manager-config\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.815362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.815337    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrdnb\" (UniqueName: \"kubernetes.io/projected/95538e94-8be7-4862-8873-75294372c415-kube-api-access-vrdnb\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.916330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.916294    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95538e94-8be7-4862-8873-75294372c415-manager-config\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.916477 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.916335    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrdnb\" (UniqueName: \"kubernetes.io/projected/95538e94-8be7-4862-8873-75294372c415-kube-api-access-vrdnb\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.916477 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.916364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95538e94-8be7-4862-8873-75294372c415-metrics-cert\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.916581 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.916493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95538e94-8be7-4862-8873-75294372c415-cert\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.917106 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.917075    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/95538e94-8be7-4862-8873-75294372c415-manager-config\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.918885 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.918856    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/95538e94-8be7-4862-8873-75294372c415-metrics-cert\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.918991 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.918950    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95538e94-8be7-4862-8873-75294372c415-cert\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:56.924683 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:56.924662    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrdnb\" (UniqueName: \"kubernetes.io/projected/95538e94-8be7-4862-8873-75294372c415-kube-api-access-vrdnb\") pod \"lws-controller-manager-6944fcc464-kk97f\" (UID: \"95538e94-8be7-4862-8873-75294372c415\") " pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:57.046820 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:57.046729    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:57.337767 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:57.337742    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"]
Sep 17 23:19:57.339726 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:19:57.339698    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95538e94_8be7_4862_8873_75294372c415.slice/crio-2ad287b1a4771a4eea37f6fd734af994b932ba212c055c17052f7b82e9d247b6 WatchSource:0}: Error finding container 2ad287b1a4771a4eea37f6fd734af994b932ba212c055c17052f7b82e9d247b6: Status 404 returned error can't find the container with id 2ad287b1a4771a4eea37f6fd734af994b932ba212c055c17052f7b82e9d247b6
Sep 17 23:19:57.729590 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:57.729509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f" event={"ID":"95538e94-8be7-4862-8873-75294372c415","Type":"ContainerStarted","Data":"2ad287b1a4771a4eea37f6fd734af994b932ba212c055c17052f7b82e9d247b6"}
Sep 17 23:19:59.740251 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:59.740216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f" event={"ID":"95538e94-8be7-4862-8873-75294372c415","Type":"ContainerStarted","Data":"abf8cfa31af7dec5d5ad63cedc38af21384da0f30554866fb699e428413b842f"}
Sep 17 23:19:59.740656 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:59.740321    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:19:59.757443 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:19:59.757392    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f" podStartSLOduration=2.394711112 podStartE2EDuration="3.757377857s" totalImagesPullingTime="1.362666745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:19:59.755920958 +0000 UTC m=+560.850447989" watchObservedRunningTime="2026-09-17 23:19:59.757377857 +0000 UTC m=+560.851904892"
Sep 17 23:20:10.747262 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:10.747230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6944fcc464-kk97f"
Sep 17 23:20:13.534088 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.534056    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"]
Sep 17 23:20:13.537522 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.537505    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.539977 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.539954    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-vnhk2\""
Sep 17 23:20:13.540110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.540021    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 23:20:13.541022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.541006    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 23:20:13.546007 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.545982    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"]
Sep 17 23:20:13.632677 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.632639    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"]
Sep 17 23:20:13.636146 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.636129    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.644331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.644304    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"]
Sep 17 23:20:13.662763 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.662723    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.662962 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.662775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msxpn\" (UniqueName: \"kubernetes.io/projected/d7f4c2c3-55ab-4eef-9c12-93deee58c041-kube-api-access-msxpn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.662962 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.662838    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.736739 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.736698    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"]
Sep 17 23:20:13.740418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.740389    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.748630 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.748598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"]
Sep 17 23:20:13.763261 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.763440 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msxpn\" (UniqueName: \"kubernetes.io/projected/d7f4c2c3-55ab-4eef-9c12-93deee58c041-kube-api-access-msxpn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.763440 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763305    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.763440 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763347    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.763440 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763371    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.763440 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763388    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lclz\" (UniqueName: \"kubernetes.io/projected/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-kube-api-access-6lclz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.763633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763612    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.763720 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.763702    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.771238 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.771207    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msxpn\" (UniqueName: \"kubernetes.io/projected/d7f4c2c3-55ab-4eef-9c12-93deee58c041-kube-api-access-msxpn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.833813 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.833773    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"]
Sep 17 23:20:13.837982 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.837960    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:13.844434 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.844404    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"]
Sep 17 23:20:13.848388 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.848364    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:13.863983 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.863942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.863983 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.863982    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lclz\" (UniqueName: \"kubernetes.io/projected/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-kube-api-access-6lclz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.864180 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.864036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.864180 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.864071    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvzh\" (UniqueName: \"kubernetes.io/projected/9c337c35-1c37-4014-b57d-0be3ae5086ef-kube-api-access-5dvzh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.864275 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.864181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.864275 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.864219    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.864691 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.864669    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.864742 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.864697    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.872318 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.872290    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lclz\" (UniqueName: \"kubernetes.io/projected/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-kube-api-access-6lclz\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.946281 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.946241    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:13.965699 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.965665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.965856 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.965705    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvzh\" (UniqueName: \"kubernetes.io/projected/9c337c35-1c37-4014-b57d-0be3ae5086ef-kube-api-access-5dvzh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.965950 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.965732    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:13.966569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.966296    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtctg\" (UniqueName: \"kubernetes.io/projected/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-kube-api-access-xtctg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:13.966569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.966372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:13.966569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.966472    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.966569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.966539    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.966969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.966947    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:13.975071 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:13.975044    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvzh\" (UniqueName: \"kubernetes.io/projected/9c337c35-1c37-4014-b57d-0be3ae5086ef-kube-api-access-5dvzh\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:14.050651 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.050611    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:14.067840 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.067812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:14.067972 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.067847    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtctg\" (UniqueName: \"kubernetes.io/projected/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-kube-api-access-xtctg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:14.067972 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.067870    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:14.068343 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.068319    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:14.068382 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.068332    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:14.077326 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.077298    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtctg\" (UniqueName: \"kubernetes.io/projected/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-kube-api-access-xtctg\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:14.129464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.129280    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"]
Sep 17 23:20:14.132189 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:20:14.132161    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f4c2c3_55ab_4eef_9c12_93deee58c041.slice/crio-d1596db698675863ed41fc4dcc883e12901a64c97d4de65fdcfb690bcad75495 WatchSource:0}: Error finding container d1596db698675863ed41fc4dcc883e12901a64c97d4de65fdcfb690bcad75495: Status 404 returned error can't find the container with id d1596db698675863ed41fc4dcc883e12901a64c97d4de65fdcfb690bcad75495
Sep 17 23:20:14.149137 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.149113    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:14.225782 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.225750    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"]
Sep 17 23:20:14.230054 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:20:14.230025    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9c8297_ccf5_4e30_bd27_8f315bb2fcb4.slice/crio-2a679c5a6fea49f993aebe040bb3ee89eebe0b3e4af9e910057c8eeb199db383 WatchSource:0}: Error finding container 2a679c5a6fea49f993aebe040bb3ee89eebe0b3e4af9e910057c8eeb199db383: Status 404 returned error can't find the container with id 2a679c5a6fea49f993aebe040bb3ee89eebe0b3e4af9e910057c8eeb199db383
Sep 17 23:20:14.324594 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.324563    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"]
Sep 17 23:20:14.326170 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:20:14.326146    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c337c35_1c37_4014_b57d_0be3ae5086ef.slice/crio-e55af01fb78ee91ec9a22aaba8d366a2a402a9a605eb92e7431578f378ac71d3 WatchSource:0}: Error finding container e55af01fb78ee91ec9a22aaba8d366a2a402a9a605eb92e7431578f378ac71d3: Status 404 returned error can't find the container with id e55af01fb78ee91ec9a22aaba8d366a2a402a9a605eb92e7431578f378ac71d3
Sep 17 23:20:14.428451 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.428420    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"]
Sep 17 23:20:14.466126 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:20:14.466095    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9231e89c_b58a_4e4b_8c61_5fdfb499c83d.slice/crio-bc90a321b8de20e1f3d825cc7c87b6b263856d036ab666343fb492e4ce7469e7 WatchSource:0}: Error finding container bc90a321b8de20e1f3d825cc7c87b6b263856d036ab666343fb492e4ce7469e7: Status 404 returned error can't find the container with id bc90a321b8de20e1f3d825cc7c87b6b263856d036ab666343fb492e4ce7469e7
Sep 17 23:20:14.823361 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.823279    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerID="fd9272b128a83d6be6a48d2025942c05633147aac2cdb7d3516e58e01a9aeaec" exitCode=0
Sep 17 23:20:14.823848 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.823374    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74" event={"ID":"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4","Type":"ContainerDied","Data":"fd9272b128a83d6be6a48d2025942c05633147aac2cdb7d3516e58e01a9aeaec"}
Sep 17 23:20:14.823848 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.823435    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74" event={"ID":"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4","Type":"ContainerStarted","Data":"2a679c5a6fea49f993aebe040bb3ee89eebe0b3e4af9e910057c8eeb199db383"}
Sep 17 23:20:14.825092 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.825072    2562 generic.go:356] "Generic (PLEG): container finished" podID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerID="96fa14cbc355ecb6fdcbb98728aa942ff69cd8dc7e9d36395a534a4740d5248f" exitCode=0
Sep 17 23:20:14.825190 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.825158    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s" event={"ID":"9231e89c-b58a-4e4b-8c61-5fdfb499c83d","Type":"ContainerDied","Data":"96fa14cbc355ecb6fdcbb98728aa942ff69cd8dc7e9d36395a534a4740d5248f"}
Sep 17 23:20:14.825252 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.825205    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s" event={"ID":"9231e89c-b58a-4e4b-8c61-5fdfb499c83d","Type":"ContainerStarted","Data":"bc90a321b8de20e1f3d825cc7c87b6b263856d036ab666343fb492e4ce7469e7"}
Sep 17 23:20:14.827205 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.827181    2562 generic.go:356] "Generic (PLEG): container finished" podID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerID="b9123370029e55a24e12eca2c86bd03e08b11d6b4ed320fceeb77531a35e5cb1" exitCode=0
Sep 17 23:20:14.827300 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.827259    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz" event={"ID":"d7f4c2c3-55ab-4eef-9c12-93deee58c041","Type":"ContainerDied","Data":"b9123370029e55a24e12eca2c86bd03e08b11d6b4ed320fceeb77531a35e5cb1"}
Sep 17 23:20:14.827349 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.827294    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz" event={"ID":"d7f4c2c3-55ab-4eef-9c12-93deee58c041","Type":"ContainerStarted","Data":"d1596db698675863ed41fc4dcc883e12901a64c97d4de65fdcfb690bcad75495"}
Sep 17 23:20:14.829091 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.829069    2562 generic.go:356] "Generic (PLEG): container finished" podID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerID="555a05922601feef5c50a33e5176be456f35c659e9103e72747269498248b63e" exitCode=0
Sep 17 23:20:14.829091 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.829104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq" event={"ID":"9c337c35-1c37-4014-b57d-0be3ae5086ef","Type":"ContainerDied","Data":"555a05922601feef5c50a33e5176be456f35c659e9103e72747269498248b63e"}
Sep 17 23:20:14.829091 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:14.829128    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq" event={"ID":"9c337c35-1c37-4014-b57d-0be3ae5086ef","Type":"ContainerStarted","Data":"e55af01fb78ee91ec9a22aaba8d366a2a402a9a605eb92e7431578f378ac71d3"}
Sep 17 23:20:15.836354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:15.836320    2562 generic.go:356] "Generic (PLEG): container finished" podID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerID="d9e261fd429f8029c01be1cbc3c4448df0bfd4bd620231d3788d2e7d893a0cd5" exitCode=0
Sep 17 23:20:15.836730 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:15.836372    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s" event={"ID":"9231e89c-b58a-4e4b-8c61-5fdfb499c83d","Type":"ContainerDied","Data":"d9e261fd429f8029c01be1cbc3c4448df0bfd4bd620231d3788d2e7d893a0cd5"}
Sep 17 23:20:16.843737 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:16.843705    2562 generic.go:356] "Generic (PLEG): container finished" podID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerID="89c8b2add18c04e0c177e63435849431fe22f211914c5b9e9044c8704dc62408" exitCode=0
Sep 17 23:20:16.844184 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:16.843796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s" event={"ID":"9231e89c-b58a-4e4b-8c61-5fdfb499c83d","Type":"ContainerDied","Data":"89c8b2add18c04e0c177e63435849431fe22f211914c5b9e9044c8704dc62408"}
Sep 17 23:20:16.845679 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:16.845653    2562 generic.go:356] "Generic (PLEG): container finished" podID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerID="a88624d1f4ec34bf53d378a9234ce68f722b2cacf52681a17c6b7dbef100a2f1" exitCode=0
Sep 17 23:20:16.845800 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:16.845689    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq" event={"ID":"9c337c35-1c37-4014-b57d-0be3ae5086ef","Type":"ContainerDied","Data":"a88624d1f4ec34bf53d378a9234ce68f722b2cacf52681a17c6b7dbef100a2f1"}
Sep 17 23:20:17.853406 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:17.853366    2562 generic.go:356] "Generic (PLEG): container finished" podID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerID="aebb5754ea152e316a9c5fca53a1f7359b0d06ee3483cf769a75878d177c2e91" exitCode=0
Sep 17 23:20:17.853812 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:17.853443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq" event={"ID":"9c337c35-1c37-4014-b57d-0be3ae5086ef","Type":"ContainerDied","Data":"aebb5754ea152e316a9c5fca53a1f7359b0d06ee3483cf769a75878d177c2e91"}
Sep 17 23:20:17.981180 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:17.981146    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:18.104015 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.103893    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-bundle\") pod \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") "
Sep 17 23:20:18.104015 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.103971    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-util\") pod \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") "
Sep 17 23:20:18.104242 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.104071    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtctg\" (UniqueName: \"kubernetes.io/projected/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-kube-api-access-xtctg\") pod \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\" (UID: \"9231e89c-b58a-4e4b-8c61-5fdfb499c83d\") "
Sep 17 23:20:18.104449 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.104417    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-bundle" (OuterVolumeSpecName: "bundle") pod "9231e89c-b58a-4e4b-8c61-5fdfb499c83d" (UID: "9231e89c-b58a-4e4b-8c61-5fdfb499c83d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:18.106278 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.106241    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-kube-api-access-xtctg" (OuterVolumeSpecName: "kube-api-access-xtctg") pod "9231e89c-b58a-4e4b-8c61-5fdfb499c83d" (UID: "9231e89c-b58a-4e4b-8c61-5fdfb499c83d"). InnerVolumeSpecName "kube-api-access-xtctg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:20:18.109040 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.109017    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-util" (OuterVolumeSpecName: "util") pod "9231e89c-b58a-4e4b-8c61-5fdfb499c83d" (UID: "9231e89c-b58a-4e4b-8c61-5fdfb499c83d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:18.205120 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.205086    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:18.205120 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.205115    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-util\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:18.205120 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.205125    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtctg\" (UniqueName: \"kubernetes.io/projected/9231e89c-b58a-4e4b-8c61-5fdfb499c83d-kube-api-access-xtctg\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:18.860482 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.860442    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s" event={"ID":"9231e89c-b58a-4e4b-8c61-5fdfb499c83d","Type":"ContainerDied","Data":"bc90a321b8de20e1f3d825cc7c87b6b263856d036ab666343fb492e4ce7469e7"}
Sep 17 23:20:18.860923 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.860505    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc90a321b8de20e1f3d825cc7c87b6b263856d036ab666343fb492e4ce7469e7"
Sep 17 23:20:18.860923 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.860463    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ccm84s"
Sep 17 23:20:18.991361 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:18.991332    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:19.112065 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.111982    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-bundle\") pod \"9c337c35-1c37-4014-b57d-0be3ae5086ef\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") "
Sep 17 23:20:19.112065 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.112025    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dvzh\" (UniqueName: \"kubernetes.io/projected/9c337c35-1c37-4014-b57d-0be3ae5086ef-kube-api-access-5dvzh\") pod \"9c337c35-1c37-4014-b57d-0be3ae5086ef\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") "
Sep 17 23:20:19.112065 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.112063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-util\") pod \"9c337c35-1c37-4014-b57d-0be3ae5086ef\" (UID: \"9c337c35-1c37-4014-b57d-0be3ae5086ef\") "
Sep 17 23:20:19.112527 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.112503    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-bundle" (OuterVolumeSpecName: "bundle") pod "9c337c35-1c37-4014-b57d-0be3ae5086ef" (UID: "9c337c35-1c37-4014-b57d-0be3ae5086ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:19.114252 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.114228    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c337c35-1c37-4014-b57d-0be3ae5086ef-kube-api-access-5dvzh" (OuterVolumeSpecName: "kube-api-access-5dvzh") pod "9c337c35-1c37-4014-b57d-0be3ae5086ef" (UID: "9c337c35-1c37-4014-b57d-0be3ae5086ef"). InnerVolumeSpecName "kube-api-access-5dvzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:20:19.117443 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.117413    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-util" (OuterVolumeSpecName: "util") pod "9c337c35-1c37-4014-b57d-0be3ae5086ef" (UID: "9c337c35-1c37-4014-b57d-0be3ae5086ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:19.212922 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.212859    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-util\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:19.213134 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.212936    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c337c35-1c37-4014-b57d-0be3ae5086ef-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:19.213134 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.212954    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvzh\" (UniqueName: \"kubernetes.io/projected/9c337c35-1c37-4014-b57d-0be3ae5086ef-kube-api-access-5dvzh\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:19.866748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.866714    2562 generic.go:356] "Generic (PLEG): container finished" podID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerID="40b7449b49996a68fbe068819f1f06f78ebd77da66bb10e458262a111d631a20" exitCode=0
Sep 17 23:20:19.867159 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.866801    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz" event={"ID":"d7f4c2c3-55ab-4eef-9c12-93deee58c041","Type":"ContainerDied","Data":"40b7449b49996a68fbe068819f1f06f78ebd77da66bb10e458262a111d631a20"}
Sep 17 23:20:19.868877 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.868863    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq"
Sep 17 23:20:19.868979 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.868852    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6fwhgq" event={"ID":"9c337c35-1c37-4014-b57d-0be3ae5086ef","Type":"ContainerDied","Data":"e55af01fb78ee91ec9a22aaba8d366a2a402a9a605eb92e7431578f378ac71d3"}
Sep 17 23:20:19.868979 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:19.868953    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e55af01fb78ee91ec9a22aaba8d366a2a402a9a605eb92e7431578f378ac71d3"
Sep 17 23:20:20.875459 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:20.875424    2562 generic.go:356] "Generic (PLEG): container finished" podID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerID="0b4ecdaff8edd57b58076ee10023eb4fcba59a6b67d6aa3a27a1541ae7090dc7" exitCode=0
Sep 17 23:20:20.875849 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:20.875507    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz" event={"ID":"d7f4c2c3-55ab-4eef-9c12-93deee58c041","Type":"ContainerDied","Data":"0b4ecdaff8edd57b58076ee10023eb4fcba59a6b67d6aa3a27a1541ae7090dc7"}
Sep 17 23:20:22.008239 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.008206    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:22.138675 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.138590    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msxpn\" (UniqueName: \"kubernetes.io/projected/d7f4c2c3-55ab-4eef-9c12-93deee58c041-kube-api-access-msxpn\") pod \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") "
Sep 17 23:20:22.138675 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.138651    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-bundle\") pod \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") "
Sep 17 23:20:22.138895 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.138683    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-util\") pod \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\" (UID: \"d7f4c2c3-55ab-4eef-9c12-93deee58c041\") "
Sep 17 23:20:22.139282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.139248    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-bundle" (OuterVolumeSpecName: "bundle") pod "d7f4c2c3-55ab-4eef-9c12-93deee58c041" (UID: "d7f4c2c3-55ab-4eef-9c12-93deee58c041"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:22.140737 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.140717    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7f4c2c3-55ab-4eef-9c12-93deee58c041-kube-api-access-msxpn" (OuterVolumeSpecName: "kube-api-access-msxpn") pod "d7f4c2c3-55ab-4eef-9c12-93deee58c041" (UID: "d7f4c2c3-55ab-4eef-9c12-93deee58c041"). InnerVolumeSpecName "kube-api-access-msxpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:20:22.143588 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.143567    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-util" (OuterVolumeSpecName: "util") pod "d7f4c2c3-55ab-4eef-9c12-93deee58c041" (UID: "d7f4c2c3-55ab-4eef-9c12-93deee58c041"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:22.239858 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.239819    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:22.239858 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.239849    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7f4c2c3-55ab-4eef-9c12-93deee58c041-util\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:22.239858 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.239858    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msxpn\" (UniqueName: \"kubernetes.io/projected/d7f4c2c3-55ab-4eef-9c12-93deee58c041-kube-api-access-msxpn\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:22.887125 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.887084    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz" event={"ID":"d7f4c2c3-55ab-4eef-9c12-93deee58c041","Type":"ContainerDied","Data":"d1596db698675863ed41fc4dcc883e12901a64c97d4de65fdcfb690bcad75495"}
Sep 17 23:20:22.887271 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.887133    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7sj5zz"
Sep 17 23:20:22.887271 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:22.887144    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1596db698675863ed41fc4dcc883e12901a64c97d4de65fdcfb690bcad75495"
Sep 17 23:20:29.925218 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:29.925111    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerID="ce83c2b9668f89987f26554bf3f40b7a9aee0e831a86e2a3e943037c737cb7d9" exitCode=0
Sep 17 23:20:29.925218 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:29.925163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74" event={"ID":"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4","Type":"ContainerDied","Data":"ce83c2b9668f89987f26554bf3f40b7a9aee0e831a86e2a3e943037c737cb7d9"}
Sep 17 23:20:30.932028 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:30.931989    2562 generic.go:356] "Generic (PLEG): container finished" podID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerID="bef94f86e611137f19d5a517c579e1c0e041f1ac5aad051e85a9226817ecbd99" exitCode=0
Sep 17 23:20:30.932520 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:30.932075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74" event={"ID":"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4","Type":"ContainerDied","Data":"bef94f86e611137f19d5a517c579e1c0e041f1ac5aad051e85a9226817ecbd99"}
Sep 17 23:20:32.062491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.062461    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:32.121282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.121239    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lclz\" (UniqueName: \"kubernetes.io/projected/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-kube-api-access-6lclz\") pod \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") "
Sep 17 23:20:32.121282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.121287    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-util\") pod \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") "
Sep 17 23:20:32.121524 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.121400    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-bundle\") pod \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\" (UID: \"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4\") "
Sep 17 23:20:32.122042 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.122017    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-bundle" (OuterVolumeSpecName: "bundle") pod "fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" (UID: "fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:32.123516 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.123492    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-kube-api-access-6lclz" (OuterVolumeSpecName: "kube-api-access-6lclz") pod "fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" (UID: "fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4"). InnerVolumeSpecName "kube-api-access-6lclz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:20:32.125877 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.125855    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-util" (OuterVolumeSpecName: "util") pod "fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" (UID: "fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:20:32.222621 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.222526    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lclz\" (UniqueName: \"kubernetes.io/projected/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-kube-api-access-6lclz\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:32.222621 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.222571    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-util\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:32.222621 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.222583    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:20:32.943951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.943893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74" event={"ID":"fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4","Type":"ContainerDied","Data":"2a679c5a6fea49f993aebe040bb3ee89eebe0b3e4af9e910057c8eeb199db383"}
Sep 17 23:20:32.943951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.943938    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ahtl74"
Sep 17 23:20:32.944165 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:32.943972    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a679c5a6fea49f993aebe040bb3ee89eebe0b3e4af9e910057c8eeb199db383"
Sep 17 23:20:53.036030 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:20:53.035991    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbd9bbc45-wsr7d"]
Sep 17 23:21:02.551463 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551376    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"]
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551762    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerName="extract"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551777    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerName="extract"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551786    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551792    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551803    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551809    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551815    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551821    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551830    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551835    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551843    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551848    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551854    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551858    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551864    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551869    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerName="pull"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551875    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerName="extract"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551880    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerName="extract"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551889    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerName="extract"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551894    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerName="extract"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551920    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551928    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerName="util"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551937    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerName="extract"
Sep 17 23:21:02.551938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.551944    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerName="extract"
Sep 17 23:21:02.552665 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.552004    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb9c8297-ccf5-4e30-bd27-8f315bb2fcb4" containerName="extract"
Sep 17 23:21:02.552665 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.552015    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9231e89c-b58a-4e4b-8c61-5fdfb499c83d" containerName="extract"
Sep 17 23:21:02.552665 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.552021    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c337c35-1c37-4014-b57d-0be3ae5086ef" containerName="extract"
Sep 17 23:21:02.552665 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.552027    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7f4c2c3-55ab-4eef-9c12-93deee58c041" containerName="extract"
Sep 17 23:21:02.555045 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.555027    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.557411 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.557385    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 23:21:02.557562 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.557430    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 23:21:02.558635 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.558612    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 23:21:02.558757 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.558614    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 23:21:02.558821 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.558621    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fpbxh\""
Sep 17 23:21:02.562491 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.562461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"]
Sep 17 23:21:02.673293 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.673230    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4efc648c-026b-447b-aac8-30f400a3d6af-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.673293 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.673297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf6n5\" (UniqueName: \"kubernetes.io/projected/4efc648c-026b-447b-aac8-30f400a3d6af-kube-api-access-rf6n5\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.673523 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.673345    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4efc648c-026b-447b-aac8-30f400a3d6af-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.774666 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.774618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4efc648c-026b-447b-aac8-30f400a3d6af-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.774666 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.774669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf6n5\" (UniqueName: \"kubernetes.io/projected/4efc648c-026b-447b-aac8-30f400a3d6af-kube-api-access-rf6n5\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.774959 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.774715    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4efc648c-026b-447b-aac8-30f400a3d6af-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.774959 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:21:02.774782    2562 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 17 23:21:02.774959 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:21:02.774859    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4efc648c-026b-447b-aac8-30f400a3d6af-plugin-serving-cert podName:4efc648c-026b-447b-aac8-30f400a3d6af nodeName:}" failed. No retries permitted until 2026-09-17 23:21:03.274838897 +0000 UTC m=+624.369365932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4efc648c-026b-447b-aac8-30f400a3d6af-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-84x5c" (UID: "4efc648c-026b-447b-aac8-30f400a3d6af") : secret "plugin-serving-cert" not found
Sep 17 23:21:02.775518 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.775493    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4efc648c-026b-447b-aac8-30f400a3d6af-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:02.783640 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:02.783611    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf6n5\" (UniqueName: \"kubernetes.io/projected/4efc648c-026b-447b-aac8-30f400a3d6af-kube-api-access-rf6n5\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:03.279745 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:03.279704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4efc648c-026b-447b-aac8-30f400a3d6af-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:03.282214 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:03.282187    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4efc648c-026b-447b-aac8-30f400a3d6af-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-84x5c\" (UID: \"4efc648c-026b-447b-aac8-30f400a3d6af\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:03.466449 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:03.466410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"
Sep 17 23:21:03.739402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:03.739369    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c"]
Sep 17 23:21:03.740561 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:21:03.740532    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4efc648c_026b_447b_aac8_30f400a3d6af.slice/crio-d6a52fa8fa24d1c9340fa36ddfe6ec86573fc336efff965df3070f2cba73457f WatchSource:0}: Error finding container d6a52fa8fa24d1c9340fa36ddfe6ec86573fc336efff965df3070f2cba73457f: Status 404 returned error can't find the container with id d6a52fa8fa24d1c9340fa36ddfe6ec86573fc336efff965df3070f2cba73457f
Sep 17 23:21:03.742119 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:03.742098    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:21:04.098949 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:04.098893    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c" event={"ID":"4efc648c-026b-447b-aac8-30f400a3d6af","Type":"ContainerStarted","Data":"d6a52fa8fa24d1c9340fa36ddfe6ec86573fc336efff965df3070f2cba73457f"}
Sep 17 23:21:09.131371 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:09.131328    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c" event={"ID":"4efc648c-026b-447b-aac8-30f400a3d6af","Type":"ContainerStarted","Data":"0dd1a4ba63c604f5bbeb32587135e81347a9719dfaf47771f86d5cdd7bfff476"}
Sep 17 23:21:09.148102 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:09.148044    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-84x5c" podStartSLOduration=2.578545957 podStartE2EDuration="7.148027319s" totalImagesPullingTime="4.569481362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:09.14718099 +0000 UTC m=+630.241708027" watchObservedRunningTime="2026-09-17 23:21:09.148027319 +0000 UTC m=+630.242554354"
Sep 17 23:21:18.055523 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.055455    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cbd9bbc45-wsr7d" podUID="b5b9eb09-84fc-4736-9039-8de7109640ea" containerName="console" containerID="cri-o://9d0edee7f7f999e76f33a2f617503f6b470f7ea4e80b1e5d3617475a73742a9f" gracePeriod=15
Sep 17 23:21:18.178633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.178601    2562 generic.go:356] "Generic (PLEG): container finished" podID="b5b9eb09-84fc-4736-9039-8de7109640ea" containerID="9d0edee7f7f999e76f33a2f617503f6b470f7ea4e80b1e5d3617475a73742a9f" exitCode=0
Sep 17 23:21:18.178755 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.178669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd9bbc45-wsr7d" event={"ID":"b5b9eb09-84fc-4736-9039-8de7109640ea","Type":"ContainerDied","Data":"9d0edee7f7f999e76f33a2f617503f6b470f7ea4e80b1e5d3617475a73742a9f"}
Sep 17 23:21:18.298022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.297998    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:21:18.420400 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420364    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-serving-cert\") pod \"b5b9eb09-84fc-4736-9039-8de7109640ea\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") "
Sep 17 23:21:18.420400 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420399    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-service-ca\") pod \"b5b9eb09-84fc-4736-9039-8de7109640ea\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") "
Sep 17 23:21:18.420647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420483    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk8tr\" (UniqueName: \"kubernetes.io/projected/b5b9eb09-84fc-4736-9039-8de7109640ea-kube-api-access-zk8tr\") pod \"b5b9eb09-84fc-4736-9039-8de7109640ea\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") "
Sep 17 23:21:18.420647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420514    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-oauth-serving-cert\") pod \"b5b9eb09-84fc-4736-9039-8de7109640ea\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") "
Sep 17 23:21:18.420647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420540    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-trusted-ca-bundle\") pod \"b5b9eb09-84fc-4736-9039-8de7109640ea\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") "
Sep 17 23:21:18.420647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420564    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-console-config\") pod \"b5b9eb09-84fc-4736-9039-8de7109640ea\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") "
Sep 17 23:21:18.420647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420590    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-oauth-config\") pod \"b5b9eb09-84fc-4736-9039-8de7109640ea\" (UID: \"b5b9eb09-84fc-4736-9039-8de7109640ea\") "
Sep 17 23:21:18.420994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.420962    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-service-ca" (OuterVolumeSpecName: "service-ca") pod "b5b9eb09-84fc-4736-9039-8de7109640ea" (UID: "b5b9eb09-84fc-4736-9039-8de7109640ea"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:18.421110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.421010    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-console-config" (OuterVolumeSpecName: "console-config") pod "b5b9eb09-84fc-4736-9039-8de7109640ea" (UID: "b5b9eb09-84fc-4736-9039-8de7109640ea"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:18.421110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.421026    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b5b9eb09-84fc-4736-9039-8de7109640ea" (UID: "b5b9eb09-84fc-4736-9039-8de7109640ea"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:18.421110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.421015    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b5b9eb09-84fc-4736-9039-8de7109640ea" (UID: "b5b9eb09-84fc-4736-9039-8de7109640ea"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:21:18.422602 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.422574    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b5b9eb09-84fc-4736-9039-8de7109640ea" (UID: "b5b9eb09-84fc-4736-9039-8de7109640ea"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:18.422689 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.422613    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b9eb09-84fc-4736-9039-8de7109640ea-kube-api-access-zk8tr" (OuterVolumeSpecName: "kube-api-access-zk8tr") pod "b5b9eb09-84fc-4736-9039-8de7109640ea" (UID: "b5b9eb09-84fc-4736-9039-8de7109640ea"). InnerVolumeSpecName "kube-api-access-zk8tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:21:18.422729 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.422707    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b5b9eb09-84fc-4736-9039-8de7109640ea" (UID: "b5b9eb09-84fc-4736-9039-8de7109640ea"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:21:18.521838 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.521799    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-oauth-serving-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:21:18.521838 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.521831    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-trusted-ca-bundle\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:21:18.521838 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.521841    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-console-config\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:21:18.522117 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.521850    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-oauth-config\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:21:18.522117 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.521861    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b5b9eb09-84fc-4736-9039-8de7109640ea-console-serving-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:21:18.522117 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.521869    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9eb09-84fc-4736-9039-8de7109640ea-service-ca\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:21:18.522117 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:18.521878    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk8tr\" (UniqueName: \"kubernetes.io/projected/b5b9eb09-84fc-4736-9039-8de7109640ea-kube-api-access-zk8tr\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:21:19.185478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:19.185451    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cbd9bbc45-wsr7d"
Sep 17 23:21:19.185478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:19.185472    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cbd9bbc45-wsr7d" event={"ID":"b5b9eb09-84fc-4736-9039-8de7109640ea","Type":"ContainerDied","Data":"108c370ec669a70ed2659653027c4faca2702817159b2c9b26989c6bd38b513f"}
Sep 17 23:21:19.185995 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:19.185528    2562 scope.go:117] "RemoveContainer" containerID="9d0edee7f7f999e76f33a2f617503f6b470f7ea4e80b1e5d3617475a73742a9f"
Sep 17 23:21:19.206269 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:19.206236    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cbd9bbc45-wsr7d"]
Sep 17 23:21:19.212363 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:19.212334    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cbd9bbc45-wsr7d"]
Sep 17 23:21:19.419294 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:19.419263    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b9eb09-84fc-4736-9039-8de7109640ea" path="/var/lib/kubelet/pods/b5b9eb09-84fc-4736-9039-8de7109640ea/volumes"
Sep 17 23:21:42.954566 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.954526    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-j96gz"]
Sep 17 23:21:42.955013 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.954855    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b9eb09-84fc-4736-9039-8de7109640ea" containerName="console"
Sep 17 23:21:42.955013 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.954867    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b9eb09-84fc-4736-9039-8de7109640ea" containerName="console"
Sep 17 23:21:42.955013 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.954948    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b9eb09-84fc-4736-9039-8de7109640ea" containerName="console"
Sep 17 23:21:42.973726 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.973686    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-j96gz"]
Sep 17 23:21:42.974093 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.973807    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:42.976299 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.976276    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 23:21:42.982889 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:42.982864    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-j96gz"]
Sep 17 23:21:43.006275 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.006238    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rxg\" (UniqueName: \"kubernetes.io/projected/cac47e37-e3a9-490b-96d7-c72a721c5671-kube-api-access-q6rxg\") pod \"limitador-limitador-84994f7ffc-j96gz\" (UID: \"cac47e37-e3a9-490b-96d7-c72a721c5671\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:43.006445 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.006293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cac47e37-e3a9-490b-96d7-c72a721c5671-config-file\") pod \"limitador-limitador-84994f7ffc-j96gz\" (UID: \"cac47e37-e3a9-490b-96d7-c72a721c5671\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:43.107055 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.107014    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rxg\" (UniqueName: \"kubernetes.io/projected/cac47e37-e3a9-490b-96d7-c72a721c5671-kube-api-access-q6rxg\") pod \"limitador-limitador-84994f7ffc-j96gz\" (UID: \"cac47e37-e3a9-490b-96d7-c72a721c5671\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:43.107257 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.107076    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cac47e37-e3a9-490b-96d7-c72a721c5671-config-file\") pod \"limitador-limitador-84994f7ffc-j96gz\" (UID: \"cac47e37-e3a9-490b-96d7-c72a721c5671\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:43.107698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.107673    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cac47e37-e3a9-490b-96d7-c72a721c5671-config-file\") pod \"limitador-limitador-84994f7ffc-j96gz\" (UID: \"cac47e37-e3a9-490b-96d7-c72a721c5671\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:43.115075 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.115045    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rxg\" (UniqueName: \"kubernetes.io/projected/cac47e37-e3a9-490b-96d7-c72a721c5671-kube-api-access-q6rxg\") pod \"limitador-limitador-84994f7ffc-j96gz\" (UID: \"cac47e37-e3a9-490b-96d7-c72a721c5671\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:43.284827 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.284738    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:43.553549 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.553454    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-wqmf6"]
Sep 17 23:21:43.558032 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.558013    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-j96gz"]
Sep 17 23:21:43.558121 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.558111    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wqmf6"
Sep 17 23:21:43.559777 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:21:43.559751    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac47e37_e3a9_490b_96d7_c72a721c5671.slice/crio-8b017a86cdb7c0c37d5b7a30f6920beae1c3ef153589a3da8c2e8dbb5af82d0f WatchSource:0}: Error finding container 8b017a86cdb7c0c37d5b7a30f6920beae1c3ef153589a3da8c2e8dbb5af82d0f: Status 404 returned error can't find the container with id 8b017a86cdb7c0c37d5b7a30f6920beae1c3ef153589a3da8c2e8dbb5af82d0f
Sep 17 23:21:43.560612 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.560577    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hvkgj\""
Sep 17 23:21:43.564011 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.563988    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wqmf6"]
Sep 17 23:21:43.610238 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.610200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4nn\" (UniqueName: \"kubernetes.io/projected/a0af09b6-91fc-4cb1-b754-e48d62221a2b-kube-api-access-hm4nn\") pod \"authorino-59c974bd48-wqmf6\" (UID: \"a0af09b6-91fc-4cb1-b754-e48d62221a2b\") " pod="kuadrant-system/authorino-59c974bd48-wqmf6"
Sep 17 23:21:43.711193 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.711158    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4nn\" (UniqueName: \"kubernetes.io/projected/a0af09b6-91fc-4cb1-b754-e48d62221a2b-kube-api-access-hm4nn\") pod \"authorino-59c974bd48-wqmf6\" (UID: \"a0af09b6-91fc-4cb1-b754-e48d62221a2b\") " pod="kuadrant-system/authorino-59c974bd48-wqmf6"
Sep 17 23:21:43.719846 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.719816    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4nn\" (UniqueName: \"kubernetes.io/projected/a0af09b6-91fc-4cb1-b754-e48d62221a2b-kube-api-access-hm4nn\") pod \"authorino-59c974bd48-wqmf6\" (UID: \"a0af09b6-91fc-4cb1-b754-e48d62221a2b\") " pod="kuadrant-system/authorino-59c974bd48-wqmf6"
Sep 17 23:21:43.876999 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:43.876964    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wqmf6"
Sep 17 23:21:44.151140 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:44.151109    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wqmf6"]
Sep 17 23:21:44.153410 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:21:44.153380    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0af09b6_91fc_4cb1_b754_e48d62221a2b.slice/crio-92fdf3b22724e8f4ba338499f3fea9c2e4ec03032a334153538052a0df0759bb WatchSource:0}: Error finding container 92fdf3b22724e8f4ba338499f3fea9c2e4ec03032a334153538052a0df0759bb: Status 404 returned error can't find the container with id 92fdf3b22724e8f4ba338499f3fea9c2e4ec03032a334153538052a0df0759bb
Sep 17 23:21:44.316434 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:44.316390    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wqmf6" event={"ID":"a0af09b6-91fc-4cb1-b754-e48d62221a2b","Type":"ContainerStarted","Data":"92fdf3b22724e8f4ba338499f3fea9c2e4ec03032a334153538052a0df0759bb"}
Sep 17 23:21:44.317937 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:44.317887    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz" event={"ID":"cac47e37-e3a9-490b-96d7-c72a721c5671","Type":"ContainerStarted","Data":"8b017a86cdb7c0c37d5b7a30f6920beae1c3ef153589a3da8c2e8dbb5af82d0f"}
Sep 17 23:21:45.325547 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:45.325509    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz" event={"ID":"cac47e37-e3a9-490b-96d7-c72a721c5671","Type":"ContainerStarted","Data":"c3c585bef15a7a78f6519cadab871a65763033590eebde53b1a86ff0c30e8d41"}
Sep 17 23:21:45.326019 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:45.325604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:21:45.342531 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:45.342476    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz" podStartSLOduration=2.410380715 podStartE2EDuration="3.342456197s" totalImagesPullingTime="932.075482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:45.341222523 +0000 UTC m=+666.435749588" watchObservedRunningTime="2026-09-17 23:21:45.342456197 +0000 UTC m=+666.436983234"
Sep 17 23:21:47.337801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:47.337762    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wqmf6" event={"ID":"a0af09b6-91fc-4cb1-b754-e48d62221a2b","Type":"ContainerStarted","Data":"8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3"}
Sep 17 23:21:47.352380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:47.352317    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-wqmf6" podStartSLOduration=1.695627566 podStartE2EDuration="4.352298334s" totalImagesPullingTime="2.656670768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:21:47.351229198 +0000 UTC m=+668.445756237" watchObservedRunningTime="2026-09-17 23:21:47.352298334 +0000 UTC m=+668.446825370"
Sep 17 23:21:56.332606 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:21:56.332573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-j96gz"
Sep 17 23:22:06.728497 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:06.728459    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wqmf6"]
Sep 17 23:22:06.728992 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:06.728645    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-wqmf6" podUID="a0af09b6-91fc-4cb1-b754-e48d62221a2b" containerName="authorino" containerID="cri-o://8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3" gracePeriod=30
Sep 17 23:22:06.981821 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:06.981757    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wqmf6"
Sep 17 23:22:07.104248 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.104208    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm4nn\" (UniqueName: \"kubernetes.io/projected/a0af09b6-91fc-4cb1-b754-e48d62221a2b-kube-api-access-hm4nn\") pod \"a0af09b6-91fc-4cb1-b754-e48d62221a2b\" (UID: \"a0af09b6-91fc-4cb1-b754-e48d62221a2b\") "
Sep 17 23:22:07.106489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.106461    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0af09b6-91fc-4cb1-b754-e48d62221a2b-kube-api-access-hm4nn" (OuterVolumeSpecName: "kube-api-access-hm4nn") pod "a0af09b6-91fc-4cb1-b754-e48d62221a2b" (UID: "a0af09b6-91fc-4cb1-b754-e48d62221a2b"). InnerVolumeSpecName "kube-api-access-hm4nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:22:07.205303 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.205257    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm4nn\" (UniqueName: \"kubernetes.io/projected/a0af09b6-91fc-4cb1-b754-e48d62221a2b-kube-api-access-hm4nn\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:22:07.440501 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.440467    2562 generic.go:356] "Generic (PLEG): container finished" podID="a0af09b6-91fc-4cb1-b754-e48d62221a2b" containerID="8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3" exitCode=0
Sep 17 23:22:07.440677 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.440527    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-wqmf6"
Sep 17 23:22:07.440677 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.440544    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wqmf6" event={"ID":"a0af09b6-91fc-4cb1-b754-e48d62221a2b","Type":"ContainerDied","Data":"8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3"}
Sep 17 23:22:07.440677 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.440584    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-wqmf6" event={"ID":"a0af09b6-91fc-4cb1-b754-e48d62221a2b","Type":"ContainerDied","Data":"92fdf3b22724e8f4ba338499f3fea9c2e4ec03032a334153538052a0df0759bb"}
Sep 17 23:22:07.440677 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.440600    2562 scope.go:117] "RemoveContainer" containerID="8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3"
Sep 17 23:22:07.449068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.449052    2562 scope.go:117] "RemoveContainer" containerID="8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3"
Sep 17 23:22:07.449362 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:22:07.449342    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3\": container with ID starting with 8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3 not found: ID does not exist" containerID="8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3"
Sep 17 23:22:07.449436 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.449372    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3"} err="failed to get container status \"8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3\": rpc error: code = NotFound desc = could not find container \"8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3\": container with ID starting with 8cb2254a337701721aa78a5829d40b6430fcc0d96eaa45ef57360031a1eb6ee3 not found: ID does not exist"
Sep 17 23:22:07.456072 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.456048    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wqmf6"]
Sep 17 23:22:07.459777 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:07.459754    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-wqmf6"]
Sep 17 23:22:09.418314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:22:09.418280    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0af09b6-91fc-4cb1-b754-e48d62221a2b" path="/var/lib/kubelet/pods/a0af09b6-91fc-4cb1-b754-e48d62221a2b/volumes"
Sep 17 23:24:07.049076 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.049036    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-d9zkc"]
Sep 17 23:24:07.049536 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.049381    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0af09b6-91fc-4cb1-b754-e48d62221a2b" containerName="authorino"
Sep 17 23:24:07.049536 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.049392    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0af09b6-91fc-4cb1-b754-e48d62221a2b" containerName="authorino"
Sep 17 23:24:07.049536 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.049462    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0af09b6-91fc-4cb1-b754-e48d62221a2b" containerName="authorino"
Sep 17 23:24:07.052445 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.052427    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.055320 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.055293    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 23:24:07.055320 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.055315    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 23:24:07.056410 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.056389    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jw2ph\""
Sep 17 23:24:07.056410 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.056404    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 23:24:07.056561 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.056426    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 23:24:07.061684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.061663    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-d9zkc"]
Sep 17 23:24:07.127110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.127068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps2wm\" (UniqueName: \"kubernetes.io/projected/d2c26a08-3d59-4e42-8254-83511fc3af92-kube-api-access-ps2wm\") pod \"s3-init-d9zkc\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") " pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.127285 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.127189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c26a08-3d59-4e42-8254-83511fc3af92-model-cache\") pod \"s3-init-d9zkc\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") " pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.228432 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.228392    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c26a08-3d59-4e42-8254-83511fc3af92-model-cache\") pod \"s3-init-d9zkc\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") " pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.228609 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.228448    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps2wm\" (UniqueName: \"kubernetes.io/projected/d2c26a08-3d59-4e42-8254-83511fc3af92-kube-api-access-ps2wm\") pod \"s3-init-d9zkc\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") " pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.228794 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.228773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c26a08-3d59-4e42-8254-83511fc3af92-model-cache\") pod \"s3-init-d9zkc\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") " pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.236993 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.236962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps2wm\" (UniqueName: \"kubernetes.io/projected/d2c26a08-3d59-4e42-8254-83511fc3af92-kube-api-access-ps2wm\") pod \"s3-init-d9zkc\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") " pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.362385 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.362350    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d9zkc"
Sep 17 23:24:07.635731 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:24:07.635698    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2c26a08_3d59_4e42_8254_83511fc3af92.slice/crio-7cd6c28a82396e5d9e665a62d6039833a313612b0bbfdbf8d846bc23a85d8151 WatchSource:0}: Error finding container 7cd6c28a82396e5d9e665a62d6039833a313612b0bbfdbf8d846bc23a85d8151: Status 404 returned error can't find the container with id 7cd6c28a82396e5d9e665a62d6039833a313612b0bbfdbf8d846bc23a85d8151
Sep 17 23:24:07.636422 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:07.636396    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-d9zkc"]
Sep 17 23:24:08.041892 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:08.041796    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d9zkc" event={"ID":"d2c26a08-3d59-4e42-8254-83511fc3af92","Type":"ContainerStarted","Data":"7cd6c28a82396e5d9e665a62d6039833a313612b0bbfdbf8d846bc23a85d8151"}
Sep 17 23:24:12.068219 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:12.068173    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d9zkc" event={"ID":"d2c26a08-3d59-4e42-8254-83511fc3af92","Type":"ContainerStarted","Data":"1ea2935d3a392b19bfcf2b69cf28cd76f0fb4902ee8155a586629f6f47f1299c"}
Sep 17 23:24:13.074851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:13.074784    2562 generic.go:356] "Generic (PLEG): container finished" podID="d2c26a08-3d59-4e42-8254-83511fc3af92" containerID="1ea2935d3a392b19bfcf2b69cf28cd76f0fb4902ee8155a586629f6f47f1299c" exitCode=0
Sep 17 23:24:13.075397 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:13.074877    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d9zkc" event={"ID":"d2c26a08-3d59-4e42-8254-83511fc3af92","Type":"ContainerDied","Data":"1ea2935d3a392b19bfcf2b69cf28cd76f0fb4902ee8155a586629f6f47f1299c"}
Sep 17 23:24:18.108653 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:18.108618    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d9zkc" event={"ID":"d2c26a08-3d59-4e42-8254-83511fc3af92","Type":"ContainerStarted","Data":"1db383c50a020d0320bcd572db4b06b5980b96e9611f98b2aef549ce88f85a27"}
Sep 17 23:24:18.126213 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:24:18.126156    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-d9zkc" podStartSLOduration=1.417373116 podStartE2EDuration="11.126137796s" totalImagesPullingTime="7.858288911s" totalInitContainerRuntime="1.850475769s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:24:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:24:18.124607646 +0000 UTC m=+819.219134694" watchObservedRunningTime="2026-09-17 23:24:18.126137796 +0000 UTC m=+819.220664857"
Sep 17 23:25:23.435842 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:23.435806    2562 generic.go:356] "Generic (PLEG): container finished" podID="d2c26a08-3d59-4e42-8254-83511fc3af92" containerID="1db383c50a020d0320bcd572db4b06b5980b96e9611f98b2aef549ce88f85a27" exitCode=0
Sep 17 23:25:23.436313 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:23.435874    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d9zkc" event={"ID":"d2c26a08-3d59-4e42-8254-83511fc3af92","Type":"ContainerDied","Data":"1db383c50a020d0320bcd572db4b06b5980b96e9611f98b2aef549ce88f85a27"}
Sep 17 23:25:24.569125 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:24.569097    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d9zkc"
Sep 17 23:25:24.701388 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:24.701274    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c26a08-3d59-4e42-8254-83511fc3af92-model-cache\") pod \"d2c26a08-3d59-4e42-8254-83511fc3af92\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") "
Sep 17 23:25:24.701388 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:24.701352    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps2wm\" (UniqueName: \"kubernetes.io/projected/d2c26a08-3d59-4e42-8254-83511fc3af92-kube-api-access-ps2wm\") pod \"d2c26a08-3d59-4e42-8254-83511fc3af92\" (UID: \"d2c26a08-3d59-4e42-8254-83511fc3af92\") "
Sep 17 23:25:24.703561 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:24.703529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c26a08-3d59-4e42-8254-83511fc3af92-kube-api-access-ps2wm" (OuterVolumeSpecName: "kube-api-access-ps2wm") pod "d2c26a08-3d59-4e42-8254-83511fc3af92" (UID: "d2c26a08-3d59-4e42-8254-83511fc3af92"). InnerVolumeSpecName "kube-api-access-ps2wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:25:24.720642 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:24.720598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c26a08-3d59-4e42-8254-83511fc3af92-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2c26a08-3d59-4e42-8254-83511fc3af92" (UID: "d2c26a08-3d59-4e42-8254-83511fc3af92"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:25:24.802318 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:24.802282    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c26a08-3d59-4e42-8254-83511fc3af92-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:25:24.802318 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:24.802311    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps2wm\" (UniqueName: \"kubernetes.io/projected/d2c26a08-3d59-4e42-8254-83511fc3af92-kube-api-access-ps2wm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:25:25.447746 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:25.447705    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-d9zkc" event={"ID":"d2c26a08-3d59-4e42-8254-83511fc3af92","Type":"ContainerDied","Data":"7cd6c28a82396e5d9e665a62d6039833a313612b0bbfdbf8d846bc23a85d8151"}
Sep 17 23:25:25.447746 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:25.447739    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-d9zkc"
Sep 17 23:25:25.447972 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:25.447748    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cd6c28a82396e5d9e665a62d6039833a313612b0bbfdbf8d846bc23a85d8151"
Sep 17 23:25:40.788451 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.788415    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"]
Sep 17 23:25:40.788798 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.788783    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c26a08-3d59-4e42-8254-83511fc3af92" containerName="download-hf-model"
Sep 17 23:25:40.788798 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.788794    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c26a08-3d59-4e42-8254-83511fc3af92" containerName="download-hf-model"
Sep 17 23:25:40.788871 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.788807    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c26a08-3d59-4e42-8254-83511fc3af92" containerName="s3-init"
Sep 17 23:25:40.788871 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.788813    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c26a08-3d59-4e42-8254-83511fc3af92" containerName="s3-init"
Sep 17 23:25:40.788871 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.788865    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c26a08-3d59-4e42-8254-83511fc3af92" containerName="s3-init"
Sep 17 23:25:40.791941 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.791924    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.795698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.795664    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 17 23:25:40.795698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.795688    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-vsxfw\""
Sep 17 23:25:40.795875 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.795729    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 17 23:25:40.795875 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.795665    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 17 23:25:40.802756 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.802735    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"]
Sep 17 23:25:40.830442 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.830407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5657522a-4c11-4bef-8216-60e693accf99-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.830442 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.830444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.830658 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.830523    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.830658 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.830568    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5fch\" (UniqueName: \"kubernetes.io/projected/5657522a-4c11-4bef-8216-60e693accf99-kube-api-access-f5fch\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.830658 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.830604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.830658 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.830619    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.931669 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.931609    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.931669 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.931668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.931872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.931781    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5657522a-4c11-4bef-8216-60e693accf99-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.931872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.931819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.932101 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.931895    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.932101 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.931967    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5fch\" (UniqueName: \"kubernetes.io/projected/5657522a-4c11-4bef-8216-60e693accf99-kube-api-access-f5fch\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.932101 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.932060    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.932264 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.932189    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.932264 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.932259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.934041 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.934015    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.934180 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.934165    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5657522a-4c11-4bef-8216-60e693accf99-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:40.939790 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:40.939758    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5fch\" (UniqueName: \"kubernetes.io/projected/5657522a-4c11-4bef-8216-60e693accf99-kube-api-access-f5fch\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:41.101845 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:41.101812    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:41.376311 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:41.376285    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"]
Sep 17 23:25:41.378127 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:25:41.378102    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5657522a_4c11_4bef_8216_60e693accf99.slice/crio-162426b87e6245461eea109d9b44ab86e1760d7116aa05480e38c3af6137825d WatchSource:0}: Error finding container 162426b87e6245461eea109d9b44ab86e1760d7116aa05480e38c3af6137825d: Status 404 returned error can't find the container with id 162426b87e6245461eea109d9b44ab86e1760d7116aa05480e38c3af6137825d
Sep 17 23:25:41.534277 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:41.534238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" event={"ID":"5657522a-4c11-4bef-8216-60e693accf99","Type":"ContainerStarted","Data":"162426b87e6245461eea109d9b44ab86e1760d7116aa05480e38c3af6137825d"}
Sep 17 23:25:43.545608 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:43.545574    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" event={"ID":"5657522a-4c11-4bef-8216-60e693accf99","Type":"ContainerStarted","Data":"fbc615c0378a44c804a3950e1b7bb7e5c7b307e85ec435c25510b07f40818bec"}
Sep 17 23:25:43.563010 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:43.562958    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" podStartSLOduration=2.10936085 podStartE2EDuration="3.562940993s" totalImagesPullingTime="1.453580143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:25:43.561601165 +0000 UTC m=+904.656128213" watchObservedRunningTime="2026-09-17 23:25:43.562940993 +0000 UTC m=+904.657468029"
Sep 17 23:25:44.102953 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:44.102916    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:51.102012 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:51.101964    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:51.114365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:51.114339    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:25:51.597478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:25:51.597450    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:26:17.343162 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:17.343126    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"]
Sep 17 23:26:17.353199 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:26:17.353155    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:26:17.353199 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:26:17.353199 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:26:17.353199 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="fbc615c0378a44c804a3950e1b7bb7e5c7b307e85ec435c25510b07f40818bec" cmd=["/bin/sleep","15"]
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:17.353232    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" message="" err=<
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:26:17.353252    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" podUID="5657522a-4c11-4bef-8216-60e693accf99" containerName="main" containerID="cri-o://fbc615c0378a44c804a3950e1b7bb7e5c7b307e85ec435c25510b07f40818bec"
Sep 17 23:26:17.353459 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:17.353332    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" podUID="5657522a-4c11-4bef-8216-60e693accf99" containerName="main" containerID="cri-o://fbc615c0378a44c804a3950e1b7bb7e5c7b307e85ec435c25510b07f40818bec" gracePeriod=60
Sep 17 23:26:17.586817 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:17.586771    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" podUID="5657522a-4c11-4bef-8216-60e693accf99" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Sep 17 23:26:17.722631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:17.722534    2562 generic.go:356] "Generic (PLEG): container finished" podID="5657522a-4c11-4bef-8216-60e693accf99" containerID="fbc615c0378a44c804a3950e1b7bb7e5c7b307e85ec435c25510b07f40818bec" exitCode=0
Sep 17 23:26:17.722631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:17.722612    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" event={"ID":"5657522a-4c11-4bef-8216-60e693accf99","Type":"ContainerDied","Data":"fbc615c0378a44c804a3950e1b7bb7e5c7b307e85ec435c25510b07f40818bec"}
Sep 17 23:26:18.094616 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.094587    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:26:18.264829 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.264787    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-home\") pod \"5657522a-4c11-4bef-8216-60e693accf99\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") "
Sep 17 23:26:18.265055 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.264850    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5fch\" (UniqueName: \"kubernetes.io/projected/5657522a-4c11-4bef-8216-60e693accf99-kube-api-access-f5fch\") pod \"5657522a-4c11-4bef-8216-60e693accf99\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") "
Sep 17 23:26:18.265055 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.264879    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5657522a-4c11-4bef-8216-60e693accf99-tls-certs\") pod \"5657522a-4c11-4bef-8216-60e693accf99\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") "
Sep 17 23:26:18.265055 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.264938    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-model-cache\") pod \"5657522a-4c11-4bef-8216-60e693accf99\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") "
Sep 17 23:26:18.265055 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.264955    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-tmp-dir\") pod \"5657522a-4c11-4bef-8216-60e693accf99\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") "
Sep 17 23:26:18.265055 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.264999    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-dshm\") pod \"5657522a-4c11-4bef-8216-60e693accf99\" (UID: \"5657522a-4c11-4bef-8216-60e693accf99\") "
Sep 17 23:26:18.265319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.265168    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-model-cache" (OuterVolumeSpecName: "model-cache") pod "5657522a-4c11-4bef-8216-60e693accf99" (UID: "5657522a-4c11-4bef-8216-60e693accf99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:18.265319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.265184    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-home" (OuterVolumeSpecName: "home") pod "5657522a-4c11-4bef-8216-60e693accf99" (UID: "5657522a-4c11-4bef-8216-60e693accf99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:18.265319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.265277    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5657522a-4c11-4bef-8216-60e693accf99" (UID: "5657522a-4c11-4bef-8216-60e693accf99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:18.267171 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.267145    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-dshm" (OuterVolumeSpecName: "dshm") pod "5657522a-4c11-4bef-8216-60e693accf99" (UID: "5657522a-4c11-4bef-8216-60e693accf99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:26:18.267269 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.267239    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5657522a-4c11-4bef-8216-60e693accf99-kube-api-access-f5fch" (OuterVolumeSpecName: "kube-api-access-f5fch") pod "5657522a-4c11-4bef-8216-60e693accf99" (UID: "5657522a-4c11-4bef-8216-60e693accf99"). InnerVolumeSpecName "kube-api-access-f5fch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:26:18.267342 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.267298    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5657522a-4c11-4bef-8216-60e693accf99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5657522a-4c11-4bef-8216-60e693accf99" (UID: "5657522a-4c11-4bef-8216-60e693accf99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:26:18.366438 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.366392    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:26:18.366438 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.366429    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:26:18.366438 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.366440    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:26:18.366438 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.366450    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5657522a-4c11-4bef-8216-60e693accf99-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:26:18.366926 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.366461    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5fch\" (UniqueName: \"kubernetes.io/projected/5657522a-4c11-4bef-8216-60e693accf99-kube-api-access-f5fch\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:26:18.366926 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.366473    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5657522a-4c11-4bef-8216-60e693accf99-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:26:18.728837 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.728809    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"
Sep 17 23:26:18.728837 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.728818    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w" event={"ID":"5657522a-4c11-4bef-8216-60e693accf99","Type":"ContainerDied","Data":"162426b87e6245461eea109d9b44ab86e1760d7116aa05480e38c3af6137825d"}
Sep 17 23:26:18.729133 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.728875    2562 scope.go:117] "RemoveContainer" containerID="fbc615c0378a44c804a3950e1b7bb7e5c7b307e85ec435c25510b07f40818bec"
Sep 17 23:26:18.751129 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.751096    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"]
Sep 17 23:26:18.754603 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:18.754576    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-xfb8w"]
Sep 17 23:26:19.418830 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:26:19.418785    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5657522a-4c11-4bef-8216-60e693accf99" path="/var/lib/kubelet/pods/5657522a-4c11-4bef-8216-60e693accf99/volumes"
Sep 17 23:27:10.395534 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.395495    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"]
Sep 17 23:27:10.396948 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.396910    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5657522a-4c11-4bef-8216-60e693accf99" containerName="main"
Sep 17 23:27:10.396948 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.396938    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5657522a-4c11-4bef-8216-60e693accf99" containerName="main"
Sep 17 23:27:10.397165 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.397136    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5657522a-4c11-4bef-8216-60e693accf99" containerName="main"
Sep 17 23:27:10.401177 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.401148    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.405220 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.404973    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 17 23:27:10.406033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.405244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 17 23:27:10.406033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.405391    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-bxngh\""
Sep 17 23:27:10.406033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.405516    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 17 23:27:10.408298 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.408267    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"]
Sep 17 23:27:10.508194 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.508152    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.508194 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.508196    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.508468 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.508218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.508468 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.508308    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxvj\" (UniqueName: \"kubernetes.io/projected/d2e3f50f-dacd-49e5-a068-ef911334b3a8-kube-api-access-5hxvj\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.508468 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.508361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.508468 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.508442    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609413 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609584 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609423    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609584 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609584 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609476    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609584 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609509    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxvj\" (UniqueName: \"kubernetes.io/projected/d2e3f50f-dacd-49e5-a068-ef911334b3a8-kube-api-access-5hxvj\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609584 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609538    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609876 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609838    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609960 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609855    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.609960 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.609943    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.611695 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.611671    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.611912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.611883    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.620459 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.620436    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxvj\" (UniqueName: \"kubernetes.io/projected/d2e3f50f-dacd-49e5-a068-ef911334b3a8-kube-api-access-5hxvj\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-sj9wr\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.714592 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.714505    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:10.984494 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.984459    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"]
Sep 17 23:27:10.985996 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:27:10.985971    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e3f50f_dacd_49e5_a068_ef911334b3a8.slice/crio-404b2e25a3393a859ac1ba199c31cc89493f287a4beb7752b5beba72146901be WatchSource:0}: Error finding container 404b2e25a3393a859ac1ba199c31cc89493f287a4beb7752b5beba72146901be: Status 404 returned error can't find the container with id 404b2e25a3393a859ac1ba199c31cc89493f287a4beb7752b5beba72146901be
Sep 17 23:27:10.996130 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:10.996104    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" event={"ID":"d2e3f50f-dacd-49e5-a068-ef911334b3a8","Type":"ContainerStarted","Data":"404b2e25a3393a859ac1ba199c31cc89493f287a4beb7752b5beba72146901be"}
Sep 17 23:27:12.003690 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:12.003647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" event={"ID":"d2e3f50f-dacd-49e5-a068-ef911334b3a8","Type":"ContainerStarted","Data":"8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62"}
Sep 17 23:27:12.020277 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:12.020181    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" podStartSLOduration=2.020166024 podStartE2EDuration="2.020166024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:27:12.018423281 +0000 UTC m=+993.112950317" watchObservedRunningTime="2026-09-17 23:27:12.020166024 +0000 UTC m=+993.114693060"
Sep 17 23:27:12.715435 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:12.715390    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:20.715179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:20.715135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:20.728039 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:20.728011    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:21.066338 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:21.066251    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:49.420856 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.420823    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"]
Sep 17 23:27:49.430995 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:27:49.430950    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:27:49.430995 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:27:49.430995 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:27:49.430995 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62" cmd=["/bin/sleep","15"]
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.431012    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" message="" err=<
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:27:49.431024    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" podUID="d2e3f50f-dacd-49e5-a068-ef911334b3a8" containerName="main" containerID="cri-o://8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62"
Sep 17 23:27:49.431336 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.431063    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" podUID="d2e3f50f-dacd-49e5-a068-ef911334b3a8" containerName="main" containerID="cri-o://8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62" gracePeriod=60
Sep 17 23:27:49.787157 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.787133    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:49.844528 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.844497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hxvj\" (UniqueName: \"kubernetes.io/projected/d2e3f50f-dacd-49e5-a068-ef911334b3a8-kube-api-access-5hxvj\") pod \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") "
Sep 17 23:27:49.844696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.844535    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-dshm\") pod \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") "
Sep 17 23:27:49.844696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.844571    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tls-certs\") pod \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") "
Sep 17 23:27:49.844696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.844615    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-model-cache\") pod \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") "
Sep 17 23:27:49.844696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.844652    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-home\") pod \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") "
Sep 17 23:27:49.844920 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.844710    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tmp-dir\") pod \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\" (UID: \"d2e3f50f-dacd-49e5-a068-ef911334b3a8\") "
Sep 17 23:27:49.844983 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.844913    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2e3f50f-dacd-49e5-a068-ef911334b3a8" (UID: "d2e3f50f-dacd-49e5-a068-ef911334b3a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:49.845042 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.845008    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-home" (OuterVolumeSpecName: "home") pod "d2e3f50f-dacd-49e5-a068-ef911334b3a8" (UID: "d2e3f50f-dacd-49e5-a068-ef911334b3a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:49.845085 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.845054    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2e3f50f-dacd-49e5-a068-ef911334b3a8" (UID: "d2e3f50f-dacd-49e5-a068-ef911334b3a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:49.846760 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.846728    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2e3f50f-dacd-49e5-a068-ef911334b3a8" (UID: "d2e3f50f-dacd-49e5-a068-ef911334b3a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:27:49.846870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.846760    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2e3f50f-dacd-49e5-a068-ef911334b3a8-kube-api-access-5hxvj" (OuterVolumeSpecName: "kube-api-access-5hxvj") pod "d2e3f50f-dacd-49e5-a068-ef911334b3a8" (UID: "d2e3f50f-dacd-49e5-a068-ef911334b3a8"). InnerVolumeSpecName "kube-api-access-5hxvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:27:49.846870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.846756    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-dshm" (OuterVolumeSpecName: "dshm") pod "d2e3f50f-dacd-49e5-a068-ef911334b3a8" (UID: "d2e3f50f-dacd-49e5-a068-ef911334b3a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:27:49.945955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.945885    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hxvj\" (UniqueName: \"kubernetes.io/projected/d2e3f50f-dacd-49e5-a068-ef911334b3a8-kube-api-access-5hxvj\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:27:49.945955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.945949    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:27:49.945955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.945959    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:27:49.945955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.945967    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:27:49.945955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.945974    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:27:49.946268 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:49.945983    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2e3f50f-dacd-49e5-a068-ef911334b3a8-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:27:50.201822 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.201786    2562 generic.go:356] "Generic (PLEG): container finished" podID="d2e3f50f-dacd-49e5-a068-ef911334b3a8" containerID="8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62" exitCode=0
Sep 17 23:27:50.202020 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.201834    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"
Sep 17 23:27:50.202020 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.201839    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" event={"ID":"d2e3f50f-dacd-49e5-a068-ef911334b3a8","Type":"ContainerDied","Data":"8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62"}
Sep 17 23:27:50.202020 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.201882    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr" event={"ID":"d2e3f50f-dacd-49e5-a068-ef911334b3a8","Type":"ContainerDied","Data":"404b2e25a3393a859ac1ba199c31cc89493f287a4beb7752b5beba72146901be"}
Sep 17 23:27:50.202020 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.201923    2562 scope.go:117] "RemoveContainer" containerID="8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62"
Sep 17 23:27:50.211915 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.211880    2562 scope.go:117] "RemoveContainer" containerID="8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62"
Sep 17 23:27:50.212236 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:27:50.212207    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62\": container with ID starting with 8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62 not found: ID does not exist" containerID="8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62"
Sep 17 23:27:50.212292 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.212246    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62"} err="failed to get container status \"8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62\": rpc error: code = NotFound desc = could not find container \"8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62\": container with ID starting with 8e1c419ef200fc5565a2a6afa71406f87984439e9c31485200364bbd5ba95b62 not found: ID does not exist"
Sep 17 23:27:50.223881 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.223854    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"]
Sep 17 23:27:50.227131 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:50.227106    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-sj9wr"]
Sep 17 23:27:51.419005 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:27:51.418969    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2e3f50f-dacd-49e5-a068-ef911334b3a8" path="/var/lib/kubelet/pods/d2e3f50f-dacd-49e5-a068-ef911334b3a8/volumes"
Sep 17 23:28:43.026633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.026600    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"]
Sep 17 23:28:43.027074 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.027007    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2e3f50f-dacd-49e5-a068-ef911334b3a8" containerName="main"
Sep 17 23:28:43.027074 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.027020    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2e3f50f-dacd-49e5-a068-ef911334b3a8" containerName="main"
Sep 17 23:28:43.027163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.027090    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2e3f50f-dacd-49e5-a068-ef911334b3a8" containerName="main"
Sep 17 23:28:43.035425 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.035397    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.038225 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.038199    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 17 23:28:43.039639 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.039619    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 17 23:28:43.039639 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.039630    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-jgn68\""
Sep 17 23:28:43.039847 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.039655    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 17 23:28:43.039847 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.039631    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-scwf8\""
Sep 17 23:28:43.045800 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.045774    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"]
Sep 17 23:28:43.188672 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.188629    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.188672 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.188671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.188887 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.188702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.188887 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.188789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.188887 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.188836    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqld7\" (UniqueName: \"kubernetes.io/projected/313de81b-9912-425a-8f84-14e67fdcf669-kube-api-access-gqld7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.189037 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.188936    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313de81b-9912-425a-8f84-14e67fdcf669-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.289937 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.289825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313de81b-9912-425a-8f84-14e67fdcf669-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.289937 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.289887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.289937 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.289931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.290207 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.289958    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.290207 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.290011    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.290207 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.290056    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqld7\" (UniqueName: \"kubernetes.io/projected/313de81b-9912-425a-8f84-14e67fdcf669-kube-api-access-gqld7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.290365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.290345    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.290425 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.290352    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.290425 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.290404    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-home\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.292300 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.292276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.292402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.292386    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313de81b-9912-425a-8f84-14e67fdcf669-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.297911 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.297874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqld7\" (UniqueName: \"kubernetes.io/projected/313de81b-9912-425a-8f84-14e67fdcf669-kube-api-access-gqld7\") pod \"scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.346444 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.346410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:43.623798 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.623769    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"]
Sep 17 23:28:43.625289 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:28:43.625263    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313de81b_9912_425a_8f84_14e67fdcf669.slice/crio-3ce6713180d0fef1bfa5b46aa48a53c500bf38ad33bd23068811f38892cd2bdb WatchSource:0}: Error finding container 3ce6713180d0fef1bfa5b46aa48a53c500bf38ad33bd23068811f38892cd2bdb: Status 404 returned error can't find the container with id 3ce6713180d0fef1bfa5b46aa48a53c500bf38ad33bd23068811f38892cd2bdb
Sep 17 23:28:43.627276 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:43.627259    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:28:44.483244 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:44.483198    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" event={"ID":"313de81b-9912-425a-8f84-14e67fdcf669","Type":"ContainerStarted","Data":"3ce6713180d0fef1bfa5b46aa48a53c500bf38ad33bd23068811f38892cd2bdb"}
Sep 17 23:28:46.497293 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:46.497258    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" event={"ID":"313de81b-9912-425a-8f84-14e67fdcf669","Type":"ContainerStarted","Data":"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"}
Sep 17 23:28:46.497698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:46.497316    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:47.503833 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:47.503793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" event={"ID":"313de81b-9912-425a-8f84-14e67fdcf669","Type":"ContainerStarted","Data":"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"}
Sep 17 23:28:47.524250 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:47.524187    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" podStartSLOduration=2.271805384 podStartE2EDuration="4.524167565s" totalImagesPullingTime="2.252362181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:28:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:28:47.522129644 +0000 UTC m=+1088.616656694" watchObservedRunningTime="2026-09-17 23:28:47.524167565 +0000 UTC m=+1088.618694648"
Sep 17 23:28:48.347175 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:48.347135    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:53.347081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:53.347048    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:53.359451 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:53.359426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:28:53.545867 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:28:53.545828    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:29:04.551547 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:04.551518    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:29:32.037229 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.037198    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"]
Sep 17 23:29:32.047346 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:29:32.047303    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:29:32.047346 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:32.047346 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:32.047346 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf" cmd=["/bin/sleep","15"]
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.047369    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" message="" err=<
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:29:32.047387    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="main" containerID="cri-o://f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"
Sep 17 23:29:32.047649 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.047437    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="main" containerID="cri-o://f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf" gracePeriod=60
Sep 17 23:29:32.264494 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.264460    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="llm-d-routing-sidecar" containerID="cri-o://86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3" gracePeriod=60
Sep 17 23:29:32.509998 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.509968    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:29:32.621246 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621208    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-home\") pod \"313de81b-9912-425a-8f84-14e67fdcf669\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") "
Sep 17 23:29:32.621246 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621251    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-model-cache\") pod \"313de81b-9912-425a-8f84-14e67fdcf669\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") "
Sep 17 23:29:32.621489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621290    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313de81b-9912-425a-8f84-14e67fdcf669-tls-certs\") pod \"313de81b-9912-425a-8f84-14e67fdcf669\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") "
Sep 17 23:29:32.621489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621312    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-dshm\") pod \"313de81b-9912-425a-8f84-14e67fdcf669\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") "
Sep 17 23:29:32.621489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621344    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-tmp-dir\") pod \"313de81b-9912-425a-8f84-14e67fdcf669\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") "
Sep 17 23:29:32.621489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621374    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqld7\" (UniqueName: \"kubernetes.io/projected/313de81b-9912-425a-8f84-14e67fdcf669-kube-api-access-gqld7\") pod \"313de81b-9912-425a-8f84-14e67fdcf669\" (UID: \"313de81b-9912-425a-8f84-14e67fdcf669\") "
Sep 17 23:29:32.621706 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621517    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-home" (OuterVolumeSpecName: "home") pod "313de81b-9912-425a-8f84-14e67fdcf669" (UID: "313de81b-9912-425a-8f84-14e67fdcf669"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:32.621706 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621612    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-model-cache" (OuterVolumeSpecName: "model-cache") pod "313de81b-9912-425a-8f84-14e67fdcf669" (UID: "313de81b-9912-425a-8f84-14e67fdcf669"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:32.621706 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.621623    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "313de81b-9912-425a-8f84-14e67fdcf669" (UID: "313de81b-9912-425a-8f84-14e67fdcf669"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:32.623637 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.623609    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-dshm" (OuterVolumeSpecName: "dshm") pod "313de81b-9912-425a-8f84-14e67fdcf669" (UID: "313de81b-9912-425a-8f84-14e67fdcf669"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:29:32.623637 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.623614    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/313de81b-9912-425a-8f84-14e67fdcf669-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "313de81b-9912-425a-8f84-14e67fdcf669" (UID: "313de81b-9912-425a-8f84-14e67fdcf669"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:29:32.623821 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.623680    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313de81b-9912-425a-8f84-14e67fdcf669-kube-api-access-gqld7" (OuterVolumeSpecName: "kube-api-access-gqld7") pod "313de81b-9912-425a-8f84-14e67fdcf669" (UID: "313de81b-9912-425a-8f84-14e67fdcf669"). InnerVolumeSpecName "kube-api-access-gqld7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:29:32.722330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.722239    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/313de81b-9912-425a-8f84-14e67fdcf669-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:29:32.722330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.722270    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:29:32.722330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.722279    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:29:32.722330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.722288    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqld7\" (UniqueName: \"kubernetes.io/projected/313de81b-9912-425a-8f84-14e67fdcf669-kube-api-access-gqld7\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:29:32.722330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.722297    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:29:32.722330 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.722305    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/313de81b-9912-425a-8f84-14e67fdcf669-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:29:32.741391 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.741356    2562 generic.go:356] "Generic (PLEG): container finished" podID="313de81b-9912-425a-8f84-14e67fdcf669" containerID="f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf" exitCode=0
Sep 17 23:29:32.741391 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.741380    2562 generic.go:356] "Generic (PLEG): container finished" podID="313de81b-9912-425a-8f84-14e67fdcf669" containerID="86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3" exitCode=0
Sep 17 23:29:32.741578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.741411    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"
Sep 17 23:29:32.741578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.741431    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" event={"ID":"313de81b-9912-425a-8f84-14e67fdcf669","Type":"ContainerDied","Data":"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"}
Sep 17 23:29:32.741578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.741486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" event={"ID":"313de81b-9912-425a-8f84-14e67fdcf669","Type":"ContainerDied","Data":"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"}
Sep 17 23:29:32.741578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.741505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm" event={"ID":"313de81b-9912-425a-8f84-14e67fdcf669","Type":"ContainerDied","Data":"3ce6713180d0fef1bfa5b46aa48a53c500bf38ad33bd23068811f38892cd2bdb"}
Sep 17 23:29:32.741578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.741525    2562 scope.go:117] "RemoveContainer" containerID="f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"
Sep 17 23:29:32.752534 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.752513    2562 scope.go:117] "RemoveContainer" containerID="86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"
Sep 17 23:29:32.760616 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.760595    2562 scope.go:117] "RemoveContainer" containerID="f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"
Sep 17 23:29:32.760954 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:29:32.760918    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf\": container with ID starting with f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf not found: ID does not exist" containerID="f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"
Sep 17 23:29:32.761085 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.760981    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"} err="failed to get container status \"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf\": rpc error: code = NotFound desc = could not find container \"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf\": container with ID starting with f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf not found: ID does not exist"
Sep 17 23:29:32.761085 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.761010    2562 scope.go:117] "RemoveContainer" containerID="86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"
Sep 17 23:29:32.761587 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:29:32.761562    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3\": container with ID starting with 86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3 not found: ID does not exist" containerID="86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"
Sep 17 23:29:32.761680 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.761590    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"} err="failed to get container status \"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3\": rpc error: code = NotFound desc = could not find container \"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3\": container with ID starting with 86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3 not found: ID does not exist"
Sep 17 23:29:32.761680 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.761613    2562 scope.go:117] "RemoveContainer" containerID="f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"
Sep 17 23:29:32.762076 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.762047    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf"} err="failed to get container status \"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf\": rpc error: code = NotFound desc = could not find container \"f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf\": container with ID starting with f4fa4b782c15187f4b5f9ce84822944d78a2a71612822091e7a8e8553f5859cf not found: ID does not exist"
Sep 17 23:29:32.762076 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.762070    2562 scope.go:117] "RemoveContainer" containerID="86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"
Sep 17 23:29:32.762340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.762321    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3"} err="failed to get container status \"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3\": rpc error: code = NotFound desc = could not find container \"86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3\": container with ID starting with 86670ae502c677302f40420f937f38329d482349ef9140e290f4ebbb5f63a3e3 not found: ID does not exist"
Sep 17 23:29:32.764256 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.764234    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"]
Sep 17 23:29:32.769533 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:32.769509    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-96b98b5c7-kblfm"]
Sep 17 23:29:32.774213 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:29:32.774193    2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313de81b_9912_425a_8f84_14e67fdcf669.slice\": RecentStats: unable to find data in memory cache]"
Sep 17 23:29:33.419045 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:33.419007    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="313de81b-9912-425a-8f84-14e67fdcf669" path="/var/lib/kubelet/pods/313de81b-9912-425a-8f84-14e67fdcf669/volumes"
Sep 17 23:29:56.410251 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.410217    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"]
Sep 17 23:29:56.410801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.410583    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="main"
Sep 17 23:29:56.410801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.410594    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="main"
Sep 17 23:29:56.410801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.410610    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="llm-d-routing-sidecar"
Sep 17 23:29:56.410801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.410616    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="llm-d-routing-sidecar"
Sep 17 23:29:56.410801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.410663    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="main"
Sep 17 23:29:56.410801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.410673    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="313de81b-9912-425a-8f84-14e67fdcf669" containerName="llm-d-routing-sidecar"
Sep 17 23:29:56.415101 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.415078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.417707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.417676    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 17 23:29:56.417884 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.417864    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 17 23:29:56.418052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.418031    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:29:56.418131 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.418095    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 17 23:29:56.418189 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.418172    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Sep 17 23:29:56.418255 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.418197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-b2xz5\""
Sep 17 23:29:56.426446 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.426422    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"]
Sep 17 23:29:56.521687 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.521649    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda339b1-6ccf-4b89-94f7-db21ee7f053a-cabundle-cert\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.521886 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.521703    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-dshm\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.521886 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.521728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tmp-dir\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.521886 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.521765    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kserve-provision-location\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.521886 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.521857    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-home\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.522132 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.521928    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9bw\" (UniqueName: \"kubernetes.io/projected/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kube-api-access-pt9bw\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.522132 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.521978    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-model-cache\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.522132 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.522003    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tls-certs\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623245 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tmp-dir\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623245 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623247    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kserve-provision-location\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623290    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-home\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623306    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9bw\" (UniqueName: \"kubernetes.io/projected/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kube-api-access-pt9bw\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-model-cache\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tls-certs\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623390    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda339b1-6ccf-4b89-94f7-db21ee7f053a-cabundle-cert\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-dshm\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623824 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623682    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-home\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623824 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623710    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kserve-provision-location\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.623960 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.623856    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tmp-dir\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.624028 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.624005    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-model-cache\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.624189 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.624170    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda339b1-6ccf-4b89-94f7-db21ee7f053a-cabundle-cert\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.625705 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.625683    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-dshm\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.625848 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.625833    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tls-certs\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.636059 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.636033    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9bw\" (UniqueName: \"kubernetes.io/projected/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kube-api-access-pt9bw\") pod \"auth-enabled-test-kserve-68df6c5db9-cfmf5\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:56.725607 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:56.725511    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:29:57.006674 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:57.006647    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"]
Sep 17 23:29:57.009350 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:29:57.009321    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda339b1_6ccf_4b89_94f7_db21ee7f053a.slice/crio-c5c743c87a821ace021d6550f36cbb20c8c7cad2e45e21dcd5842cec730a3819 WatchSource:0}: Error finding container c5c743c87a821ace021d6550f36cbb20c8c7cad2e45e21dcd5842cec730a3819: Status 404 returned error can't find the container with id c5c743c87a821ace021d6550f36cbb20c8c7cad2e45e21dcd5842cec730a3819
Sep 17 23:29:57.870309 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:57.870268    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" event={"ID":"fda339b1-6ccf-4b89-94f7-db21ee7f053a","Type":"ContainerStarted","Data":"c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e"}
Sep 17 23:29:57.870675 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:57.870332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" event={"ID":"fda339b1-6ccf-4b89-94f7-db21ee7f053a","Type":"ContainerStarted","Data":"c5c743c87a821ace021d6550f36cbb20c8c7cad2e45e21dcd5842cec730a3819"}
Sep 17 23:29:59.884948 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:59.884889    2562 generic.go:356] "Generic (PLEG): container finished" podID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerID="c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e" exitCode=0
Sep 17 23:29:59.885395 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:29:59.884962    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" event={"ID":"fda339b1-6ccf-4b89-94f7-db21ee7f053a","Type":"ContainerDied","Data":"c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e"}
Sep 17 23:30:28.073003 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:30:28.072961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" event={"ID":"fda339b1-6ccf-4b89-94f7-db21ee7f053a","Type":"ContainerStarted","Data":"00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63"}
Sep 17 23:30:28.096540 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:30:28.096483    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podStartSLOduration=2.316886719 podStartE2EDuration="32.096466665s" totalImagesPullingTime="27.574562558s" totalInitContainerRuntime="2.205017388s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:30:28.094555958 +0000 UTC m=+1189.189082993" watchObservedRunningTime="2026-09-17 23:30:28.096466665 +0000 UTC m=+1189.190993701"
Sep 17 23:30:28.725972 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:30:28.725924    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:30:36.726388 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:30:36.726344    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:30:36.728015 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:30:36.727981    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:30:46.726443 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:30:46.726392    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:30:56.726950 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:30:56.726892    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:31:06.726048 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:31:06.725997    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:31:16.726375 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:31:16.726325    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:31:26.726797 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:31:26.726741    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:31:36.726289 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:31:36.726198    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:31:46.726366 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:31:46.726304    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:31:56.726809 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:31:56.726761    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:32:06.726179 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:06.726129    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:32:16.336804 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.336771    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"]
Sep 17 23:32:16.341328 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.341295    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.343674 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.343648    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:32:16.344237 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.344215    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Sep 17 23:32:16.344351 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.344317    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 17 23:32:16.344351 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.344320    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 17 23:32:16.345084 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.345061    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 17 23:32:16.345424 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.345406    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-rbm66\""
Sep 17 23:32:16.353034 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.353013    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"]
Sep 17 23:32:16.455421 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455380    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-dshm\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.455421 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-model-cache\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.455633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.455633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmdsv\" (UniqueName: \"kubernetes.io/projected/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kube-api-access-bmdsv\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.455633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455499    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/142188c7-151e-4d8c-9ab9-ba69a6af7c48-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.455633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455525    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-home\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.455633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455546    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.455633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.455567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.556726 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.556692    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.556726 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.556729    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmdsv\" (UniqueName: \"kubernetes.io/projected/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kube-api-access-bmdsv\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.556889 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.556756    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/142188c7-151e-4d8c-9ab9-ba69a6af7c48-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557024 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.556942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-home\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557024 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.556982    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557024 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.557021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557184 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.557163    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-dshm\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557308 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.557214    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-model-cache\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557690 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.557655    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/142188c7-151e-4d8c-9ab9-ba69a6af7c48-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557816 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.557778    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-home\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557880 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.557812    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.557980 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.557962    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.558081 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.558057    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-model-cache\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.559650 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.559631    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-dshm\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.559809 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.559786    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.569751 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.569721    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmdsv\" (UniqueName: \"kubernetes.io/projected/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kube-api-access-bmdsv\") pod \"tokenizer-clean-path-test-kserve-74d4865dfc-nb976\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.653492 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.653408    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:16.727079 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.726659    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": dial tcp 10.133.0.43:8000: connect: connection refused"
Sep 17 23:32:16.937915 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:16.937863    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"]
Sep 17 23:32:16.939120 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:32:16.939095    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod142188c7_151e_4d8c_9ab9_ba69a6af7c48.slice/crio-7a3c9055769caa9368934ee535b276b7b87fe1b6afe1089b922c27603f18c63c WatchSource:0}: Error finding container 7a3c9055769caa9368934ee535b276b7b87fe1b6afe1089b922c27603f18c63c: Status 404 returned error can't find the container with id 7a3c9055769caa9368934ee535b276b7b87fe1b6afe1089b922c27603f18c63c
Sep 17 23:32:17.676602 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:17.676552    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" event={"ID":"142188c7-151e-4d8c-9ab9-ba69a6af7c48","Type":"ContainerStarted","Data":"71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581"}
Sep 17 23:32:17.677106 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:17.676619    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" event={"ID":"142188c7-151e-4d8c-9ab9-ba69a6af7c48","Type":"ContainerStarted","Data":"7a3c9055769caa9368934ee535b276b7b87fe1b6afe1089b922c27603f18c63c"}
Sep 17 23:32:19.690114 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:19.690027    2562 generic.go:356] "Generic (PLEG): container finished" podID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerID="71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581" exitCode=0
Sep 17 23:32:19.690114 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:19.690097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" event={"ID":"142188c7-151e-4d8c-9ab9-ba69a6af7c48","Type":"ContainerDied","Data":"71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581"}
Sep 17 23:32:20.697268 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:20.697228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" event={"ID":"142188c7-151e-4d8c-9ab9-ba69a6af7c48","Type":"ContainerStarted","Data":"1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1"}
Sep 17 23:32:20.719872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:20.719805    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" podStartSLOduration=2.310780372 podStartE2EDuration="4.719789882s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.40900951s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:32:20.716928581 +0000 UTC m=+1301.811455612" watchObservedRunningTime="2026-09-17 23:32:20.719789882 +0000 UTC m=+1301.814316917"
Sep 17 23:32:21.653754 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:21.653704    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:26.654507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:26.654468    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:26.667402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:26.667379    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:26.737890 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:26.737861    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:32:26.741097 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:26.741070    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:32:26.745826 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:26.745805    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:32:39.722886 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:39.722849    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"]
Sep 17 23:32:54.747322 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:54.747275    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main" containerID="cri-o://00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63" gracePeriod=45
Sep 17 23:32:55.505374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.505342    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:32:55.597862 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.597831    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda339b1-6ccf-4b89-94f7-db21ee7f053a-cabundle-cert\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598064 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.597888    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-dshm\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598113 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598089    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-home\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598159 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598126    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kserve-provision-location\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598206 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598189    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tls-certs\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598263 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598206    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda339b1-6ccf-4b89-94f7-db21ee7f053a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:32:55.598263 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598230    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt9bw\" (UniqueName: \"kubernetes.io/projected/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kube-api-access-pt9bw\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598369 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-model-cache\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598369 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598309    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tmp-dir\") pod \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\" (UID: \"fda339b1-6ccf-4b89-94f7-db21ee7f053a\") "
Sep 17 23:32:55.598617 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598586    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-model-cache" (OuterVolumeSpecName: "model-cache") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:55.598617 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598604    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fda339b1-6ccf-4b89-94f7-db21ee7f053a-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.598804 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.598781    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-home" (OuterVolumeSpecName: "home") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:55.600673 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.600640    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-dshm" (OuterVolumeSpecName: "dshm") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:55.600791 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.600727    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kube-api-access-pt9bw" (OuterVolumeSpecName: "kube-api-access-pt9bw") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "kube-api-access-pt9bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:32:55.600791 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.600740    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:32:55.612234 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.612164    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:55.619597 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.619557    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fda339b1-6ccf-4b89-94f7-db21ee7f053a" (UID: "fda339b1-6ccf-4b89-94f7-db21ee7f053a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:32:55.699838 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.699805    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.699838 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.699836    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.699838 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.699845    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.700131 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.699855    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.700131 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.699865    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt9bw\" (UniqueName: \"kubernetes.io/projected/fda339b1-6ccf-4b89-94f7-db21ee7f053a-kube-api-access-pt9bw\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.700131 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.699873    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.700131 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.699881    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fda339b1-6ccf-4b89-94f7-db21ee7f053a-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:32:55.890064 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.889979    2562 generic.go:356] "Generic (PLEG): container finished" podID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerID="00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63" exitCode=0
Sep 17 23:32:55.890064 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.890039    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"
Sep 17 23:32:55.890064 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.890042    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" event={"ID":"fda339b1-6ccf-4b89-94f7-db21ee7f053a","Type":"ContainerDied","Data":"00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63"}
Sep 17 23:32:55.890532 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.890097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5" event={"ID":"fda339b1-6ccf-4b89-94f7-db21ee7f053a","Type":"ContainerDied","Data":"c5c743c87a821ace021d6550f36cbb20c8c7cad2e45e21dcd5842cec730a3819"}
Sep 17 23:32:55.890532 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.890127    2562 scope.go:117] "RemoveContainer" containerID="00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63"
Sep 17 23:32:55.900098 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.900074    2562 scope.go:117] "RemoveContainer" containerID="c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e"
Sep 17 23:32:55.908437 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.908416    2562 scope.go:117] "RemoveContainer" containerID="00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63"
Sep 17 23:32:55.908689 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:32:55.908671    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63\": container with ID starting with 00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63 not found: ID does not exist" containerID="00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63"
Sep 17 23:32:55.908733 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.908696    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63"} err="failed to get container status \"00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63\": rpc error: code = NotFound desc = could not find container \"00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63\": container with ID starting with 00f0ce6e861f045710504d50112e667ff35fcfe20ee5d087563b669f49f9ab63 not found: ID does not exist"
Sep 17 23:32:55.908733 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.908712    2562 scope.go:117] "RemoveContainer" containerID="c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e"
Sep 17 23:32:55.909005 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:32:55.908986    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e\": container with ID starting with c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e not found: ID does not exist" containerID="c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e"
Sep 17 23:32:55.909069 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.909011    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e"} err="failed to get container status \"c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e\": rpc error: code = NotFound desc = could not find container \"c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e\": container with ID starting with c40fdcd9e3aa0c72eb8bfaad567839530356fdf5a2e2262ca3e766e80d005c2e not found: ID does not exist"
Sep 17 23:32:55.914044 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.914022    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"]
Sep 17 23:32:55.919243 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:55.919219    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-68df6c5db9-cfmf5"]
Sep 17 23:32:57.418371 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:32:57.418336    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" path="/var/lib/kubelet/pods/fda339b1-6ccf-4b89-94f7-db21ee7f053a/volumes"
Sep 17 23:33:32.342692 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.342655    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"]
Sep 17 23:33:32.343175 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.343059    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main"
Sep 17 23:33:32.343175 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.343071    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main"
Sep 17 23:33:32.343175 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.343083    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="storage-initializer"
Sep 17 23:33:32.343175 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.343090    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="storage-initializer"
Sep 17 23:33:32.343175 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.343148    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda339b1-6ccf-4b89-94f7-db21ee7f053a" containerName="main"
Sep 17 23:33:32.346426 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.346410    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.349624 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.349597    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"seaweedfs-s3-creds\""
Sep 17 23:33:32.349756 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.349731    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:33:32.352645 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.352624    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 17 23:33:32.352786 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.352648    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 17 23:33:32.352786 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.352679    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-fcl7d\""
Sep 17 23:33:32.352786 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.352648    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 17 23:33:32.374193 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.374164    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"]
Sep 17 23:33:32.406861 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.406829    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tls-certs\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.407049 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.406881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.407049 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.406931    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tz9k\" (UniqueName: \"kubernetes.io/projected/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kube-api-access-8tz9k\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.407049 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.406968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-model-cache\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.407049 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.407040    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-home\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.407194 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.407066    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-dshm\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.407194 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.407086    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf5fe7-9177-4bba-9c37-dafa009f2969-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.407194 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.407140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tmp-dir\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508300 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508265    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tmp-dir\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508300 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508301    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tls-certs\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508483    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tz9k\" (UniqueName: \"kubernetes.io/projected/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kube-api-access-8tz9k\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508569 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508539    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-model-cache\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508738 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508586    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-home\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508738 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508611    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-dshm\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508738 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf5fe7-9177-4bba-9c37-dafa009f2969-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508889 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508748    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tmp-dir\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.508988 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508890    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-model-cache\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.509043 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.508981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-home\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.509115 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.509091    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.509420 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.509398    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf5fe7-9177-4bba-9c37-dafa009f2969-cabundle-cert\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.511204 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.511188    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-dshm\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.511590 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.511569    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tls-certs\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.516124 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.516104    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tz9k\" (UniqueName: \"kubernetes.io/projected/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kube-api-access-8tz9k\") pod \"auth-invalid-token-test-kserve-6899f4fc85-knkwh\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.657173 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.657078    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:32.935572 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:32.935538    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"]
Sep 17 23:33:33.078026 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:33.077986    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" event={"ID":"bbbf5fe7-9177-4bba-9c37-dafa009f2969","Type":"ContainerStarted","Data":"4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683"}
Sep 17 23:33:33.078202 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:33.078084    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" event={"ID":"bbbf5fe7-9177-4bba-9c37-dafa009f2969","Type":"ContainerStarted","Data":"6a7448f02cfdd830b6d06d2b2d606fe1acbb7af525bc68081d6c10b859c4e6ad"}
Sep 17 23:33:36.097278 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:36.097246    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerID="4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683" exitCode=0
Sep 17 23:33:36.097666 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:36.097300    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" event={"ID":"bbbf5fe7-9177-4bba-9c37-dafa009f2969","Type":"ContainerDied","Data":"4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683"}
Sep 17 23:33:37.104777 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:37.104743    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" event={"ID":"bbbf5fe7-9177-4bba-9c37-dafa009f2969","Type":"ContainerStarted","Data":"3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175"}
Sep 17 23:33:37.129226 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:37.129148    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podStartSLOduration=2.866925531 podStartE2EDuration="5.129125386s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.262199855s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:33:37.125129878 +0000 UTC m=+1378.219656914" watchObservedRunningTime="2026-09-17 23:33:37.129125386 +0000 UTC m=+1378.223652425"
Sep 17 23:33:37.657748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:37.657701    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:42.657363 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:42.657324    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:33:42.658827 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:42.658796    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:33:45.179579 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.179539    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"]
Sep 17 23:33:45.189222 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:33:45.189180    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:33:45.189222 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:33:45.189222 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:33:45.189222 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1" cmd=["/bin/sleep","15"]
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.189240    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" message="" err=<
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:33:45.189251    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerName="main" containerID="cri-o://1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1"
Sep 17 23:33:45.189538 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.189289    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerName="main" containerID="cri-o://1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1" gracePeriod=60
Sep 17 23:33:45.547065 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.547031    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:33:45.630125 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630083    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-model-cache\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630125 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630127    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tmp-dir\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630154    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-dshm\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630205    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tls-certs\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630223    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmdsv\" (UniqueName: \"kubernetes.io/projected/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kube-api-access-bmdsv\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630245    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/142188c7-151e-4d8c-9ab9-ba69a6af7c48-cabundle-cert\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630359    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kserve-provision-location\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630648 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630382    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-home\") pod \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\" (UID: \"142188c7-151e-4d8c-9ab9-ba69a6af7c48\") "
Sep 17 23:33:45.630648 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630419    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-model-cache" (OuterVolumeSpecName: "model-cache") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.630648 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630434    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.630807 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630648    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.630807 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630664    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.630807 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630792    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-home" (OuterVolumeSpecName: "home") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.630989 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.630824    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/142188c7-151e-4d8c-9ab9-ba69a6af7c48-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:33:45.632633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.632601    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-dshm" (OuterVolumeSpecName: "dshm") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.632633 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.632615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:33:45.632840 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.632825    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kube-api-access-bmdsv" (OuterVolumeSpecName: "kube-api-access-bmdsv") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "kube-api-access-bmdsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:33:45.650962 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.650927    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "142188c7-151e-4d8c-9ab9-ba69a6af7c48" (UID: "142188c7-151e-4d8c-9ab9-ba69a6af7c48"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:33:45.731758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.731679    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmdsv\" (UniqueName: \"kubernetes.io/projected/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kube-api-access-bmdsv\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.731758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.731707    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/142188c7-151e-4d8c-9ab9-ba69a6af7c48-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.731758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.731717    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.731758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.731731    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.731758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.731741    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/142188c7-151e-4d8c-9ab9-ba69a6af7c48-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:45.731758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:45.731748    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/142188c7-151e-4d8c-9ab9-ba69a6af7c48-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:33:46.161382 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.161349    2562 generic.go:356] "Generic (PLEG): container finished" podID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerID="1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1" exitCode=0
Sep 17 23:33:46.161562 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.161415    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"
Sep 17 23:33:46.161562 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.161432    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" event={"ID":"142188c7-151e-4d8c-9ab9-ba69a6af7c48","Type":"ContainerDied","Data":"1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1"}
Sep 17 23:33:46.161562 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.161485    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976" event={"ID":"142188c7-151e-4d8c-9ab9-ba69a6af7c48","Type":"ContainerDied","Data":"7a3c9055769caa9368934ee535b276b7b87fe1b6afe1089b922c27603f18c63c"}
Sep 17 23:33:46.161562 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.161508    2562 scope.go:117] "RemoveContainer" containerID="1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1"
Sep 17 23:33:46.170786 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.170766    2562 scope.go:117] "RemoveContainer" containerID="71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581"
Sep 17 23:33:46.178994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.178975    2562 scope.go:117] "RemoveContainer" containerID="1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1"
Sep 17 23:33:46.179249 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:33:46.179231    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1\": container with ID starting with 1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1 not found: ID does not exist" containerID="1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1"
Sep 17 23:33:46.179295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.179258    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1"} err="failed to get container status \"1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1\": rpc error: code = NotFound desc = could not find container \"1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1\": container with ID starting with 1150a11cef0932603fb9ead152f3882adf7ed099d87a00fd11c695a8f79c70c1 not found: ID does not exist"
Sep 17 23:33:46.179295 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.179274    2562 scope.go:117] "RemoveContainer" containerID="71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581"
Sep 17 23:33:46.179496 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:33:46.179481    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581\": container with ID starting with 71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581 not found: ID does not exist" containerID="71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581"
Sep 17 23:33:46.179540 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.179501    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581"} err="failed to get container status \"71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581\": rpc error: code = NotFound desc = could not find container \"71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581\": container with ID starting with 71039c51d5d150a00f548ca8e1c362d754e10a8dfb2b9fc19146f2bab0d77581 not found: ID does not exist"
Sep 17 23:33:46.187319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.187292    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"]
Sep 17 23:33:46.191326 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:46.191303    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74d4865dfc-nb976"]
Sep 17 23:33:47.420223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:47.420189    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" path="/var/lib/kubelet/pods/142188c7-151e-4d8c-9ab9-ba69a6af7c48/volumes"
Sep 17 23:33:52.657697 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:33:52.657658    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:34:02.657962 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:02.657883    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:34:12.658214 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:12.658170    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:34:22.658120 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:22.658081    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:34:32.657930 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:32.657814    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:34:41.096405 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.096365    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"]
Sep 17 23:34:41.096842 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.096820    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerName="main"
Sep 17 23:34:41.096842 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.096836    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerName="main"
Sep 17 23:34:41.096978 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.096851    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerName="storage-initializer"
Sep 17 23:34:41.096978 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.096856    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerName="storage-initializer"
Sep 17 23:34:41.096978 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.096941    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="142188c7-151e-4d8c-9ab9-ba69a6af7c48" containerName="main"
Sep 17 23:34:41.100107 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.100090    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.102871 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.102847    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Sep 17 23:34:41.103037 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.102887    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 17 23:34:41.104390 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.104373    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 17 23:34:41.104619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.104597    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:34:41.104736 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.104718    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 17 23:34:41.104841 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.104819    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-2qblg\""
Sep 17 23:34:41.110646 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.110624    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"]
Sep 17 23:34:41.241630 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.241806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241637    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-dshm\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.241806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhzkw\" (UniqueName: \"kubernetes.io/projected/34a98844-0963-45aa-8c9e-073bb98c592d-kube-api-access-vhzkw\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.241806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a98844-0963-45aa-8c9e-073bb98c592d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.241806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.241806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241766    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a98844-0963-45aa-8c9e-073bb98c592d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.241806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241789    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-home\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.241806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.241803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.342983 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.342945    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343165 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.342996    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-dshm\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343165 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343136    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhzkw\" (UniqueName: \"kubernetes.io/projected/34a98844-0963-45aa-8c9e-073bb98c592d-kube-api-access-vhzkw\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343207    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a98844-0963-45aa-8c9e-073bb98c592d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343388 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343312    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a98844-0963-45aa-8c9e-073bb98c592d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343388 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-home\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343388 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343382    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343557 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343392    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-model-cache\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343797 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343773    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.343993 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343971    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.344139 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.344018    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a98844-0963-45aa-8c9e-073bb98c592d-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.344139 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.343941    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-home\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.345510 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.345483    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-dshm\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.345894 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.345869    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a98844-0963-45aa-8c9e-073bb98c592d-tls-certs\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.351469 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.351416    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhzkw\" (UniqueName: \"kubernetes.io/projected/34a98844-0963-45aa-8c9e-073bb98c592d-kube-api-access-vhzkw\") pod \"tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.411234 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.411196    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:41.687594 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:41.687563    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"]
Sep 17 23:34:41.690008 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:34:41.689964    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a98844_0963_45aa_8c9e_073bb98c592d.slice/crio-95fe7f8c9e36b8117fd3605d2307e2a51d204b62422c794e1253a3d59da3abbd WatchSource:0}: Error finding container 95fe7f8c9e36b8117fd3605d2307e2a51d204b62422c794e1253a3d59da3abbd: Status 404 returned error can't find the container with id 95fe7f8c9e36b8117fd3605d2307e2a51d204b62422c794e1253a3d59da3abbd
Sep 17 23:34:42.495763 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:42.495719    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" event={"ID":"34a98844-0963-45aa-8c9e-073bb98c592d","Type":"ContainerStarted","Data":"2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d"}
Sep 17 23:34:42.496252 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:42.495792    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" event={"ID":"34a98844-0963-45aa-8c9e-073bb98c592d","Type":"ContainerStarted","Data":"95fe7f8c9e36b8117fd3605d2307e2a51d204b62422c794e1253a3d59da3abbd"}
Sep 17 23:34:42.658178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:42.658130    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:34:44.510199 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:44.510110    2562 generic.go:356] "Generic (PLEG): container finished" podID="34a98844-0963-45aa-8c9e-073bb98c592d" containerID="2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d" exitCode=0
Sep 17 23:34:44.510199 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:44.510163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" event={"ID":"34a98844-0963-45aa-8c9e-073bb98c592d","Type":"ContainerDied","Data":"2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d"}
Sep 17 23:34:45.518667 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:45.518631    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" event={"ID":"34a98844-0963-45aa-8c9e-073bb98c592d","Type":"ContainerStarted","Data":"5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39"}
Sep 17 23:34:45.538040 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:45.537989    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" podStartSLOduration=2.145712665 podStartE2EDuration="4.537971452s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.392258787s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:34:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:34:45.537877811 +0000 UTC m=+1446.632404847" watchObservedRunningTime="2026-09-17 23:34:45.537971452 +0000 UTC m=+1446.632498489"
Sep 17 23:34:46.412287 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:46.412243    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:51.411472 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:51.411433    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:51.425272 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:51.425246    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:51.565174 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:51.565142    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:34:52.658546 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:34:52.658502    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:35:02.658028 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:02.657982    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:35:12.657652 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:12.657595    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:35:18.953149 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:18.953113    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"]
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:35:18.962524    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39" cmd=["/bin/sleep","15"]
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:18.962587    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" message="" err=<
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:35:18.962604    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:35:18.962667 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" containerName="main" containerID="cri-o://5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39"
Sep 17 23:35:18.963378 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:18.962665    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" containerName="main" containerID="cri-o://5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39" gracePeriod=60
Sep 17 23:35:19.317524 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.317489    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:35:19.387544 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387499    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhzkw\" (UniqueName: \"kubernetes.io/projected/34a98844-0963-45aa-8c9e-073bb98c592d-kube-api-access-vhzkw\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.387745 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387578    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-kserve-provision-location\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.387745 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387608    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-dshm\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.387745 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387641    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a98844-0963-45aa-8c9e-073bb98c592d-tls-certs\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.387745 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387685    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a98844-0963-45aa-8c9e-073bb98c592d-cabundle-cert\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.387996 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387773    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-tmp-dir\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.387996 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387828    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-home\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.387996 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.387865    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-model-cache\") pod \"34a98844-0963-45aa-8c9e-073bb98c592d\" (UID: \"34a98844-0963-45aa-8c9e-073bb98c592d\") "
Sep 17 23:35:19.388155 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.388082    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-home" (OuterVolumeSpecName: "home") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.388222 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.388183    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.388315 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.388293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-model-cache" (OuterVolumeSpecName: "model-cache") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.388407 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.388375    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.388582 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.388558    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a98844-0963-45aa-8c9e-073bb98c592d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:19.390366 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.390336    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34a98844-0963-45aa-8c9e-073bb98c592d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:19.390483 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.390378    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-dshm" (OuterVolumeSpecName: "dshm") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.390483 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.390402    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a98844-0963-45aa-8c9e-073bb98c592d-kube-api-access-vhzkw" (OuterVolumeSpecName: "kube-api-access-vhzkw") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "kube-api-access-vhzkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:35:19.415279 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.415236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34a98844-0963-45aa-8c9e-073bb98c592d" (UID: "34a98844-0963-45aa-8c9e-073bb98c592d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:19.489331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.489225    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.489331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.489256    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.489331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.489264    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhzkw\" (UniqueName: \"kubernetes.io/projected/34a98844-0963-45aa-8c9e-073bb98c592d-kube-api-access-vhzkw\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.489331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.489273    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.489331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.489281    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34a98844-0963-45aa-8c9e-073bb98c592d-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.489331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.489289    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34a98844-0963-45aa-8c9e-073bb98c592d-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.489331 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.489296    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34a98844-0963-45aa-8c9e-073bb98c592d-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:19.709857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.709813    2562 generic.go:356] "Generic (PLEG): container finished" podID="34a98844-0963-45aa-8c9e-073bb98c592d" containerID="5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39" exitCode=0
Sep 17 23:35:19.710068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.709894    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" event={"ID":"34a98844-0963-45aa-8c9e-073bb98c592d","Type":"ContainerDied","Data":"5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39"}
Sep 17 23:35:19.710068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.709922    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"
Sep 17 23:35:19.710068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.710003    2562 scope.go:117] "RemoveContainer" containerID="5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39"
Sep 17 23:35:19.710211 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.709992    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t" event={"ID":"34a98844-0963-45aa-8c9e-073bb98c592d","Type":"ContainerDied","Data":"95fe7f8c9e36b8117fd3605d2307e2a51d204b62422c794e1253a3d59da3abbd"}
Sep 17 23:35:19.721274 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.721241    2562 scope.go:117] "RemoveContainer" containerID="2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d"
Sep 17 23:35:19.727677 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.727651    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"]
Sep 17 23:35:19.729916 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.729882    2562 scope.go:117] "RemoveContainer" containerID="5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39"
Sep 17 23:35:19.730232 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:35:19.730209    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39\": container with ID starting with 5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39 not found: ID does not exist" containerID="5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39"
Sep 17 23:35:19.730319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.730239    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39"} err="failed to get container status \"5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39\": rpc error: code = NotFound desc = could not find container \"5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39\": container with ID starting with 5f7dfd3a4c68c92a7b90d08637186ff79ec3b3060dea1b698a0a6ee541022c39 not found: ID does not exist"
Sep 17 23:35:19.730319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.730258    2562 scope.go:117] "RemoveContainer" containerID="2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d"
Sep 17 23:35:19.730659 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:35:19.730614    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d\": container with ID starting with 2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d not found: ID does not exist" containerID="2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d"
Sep 17 23:35:19.730748 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.730651    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d"} err="failed to get container status \"2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d\": rpc error: code = NotFound desc = could not find container \"2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d\": container with ID starting with 2578dd4d9eece158d645f69e13a5c70fe3634e2f61a074d251157f577f7ab25d not found: ID does not exist"
Sep 17 23:35:19.733707 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:19.733685    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-7f5b8b89cb-km67t"]
Sep 17 23:35:21.418948 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:21.418896    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" path="/var/lib/kubelet/pods/34a98844-0963-45aa-8c9e-073bb98c592d/volumes"
Sep 17 23:35:22.657737 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:22.657694    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:35:32.668414 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:32.668383    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:35:32.676574 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:32.676548    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:35:34.329681 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:34.329641    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"]
Sep 17 23:35:49.353248 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:49.353181    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" containerID="cri-o://3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175" gracePeriod=45
Sep 17 23:35:49.669915 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:49.669796    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Sep 17 23:35:50.108353 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.108321    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:35:50.155735 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.155694    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tz9k\" (UniqueName: \"kubernetes.io/projected/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kube-api-access-8tz9k\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.155933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.155741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf5fe7-9177-4bba-9c37-dafa009f2969-cabundle-cert\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.155933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.155810    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-model-cache\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.155933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.155839    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tls-certs\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.155933 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.155869    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-home\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.156158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.155935    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-dshm\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.156158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.155965    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kserve-provision-location\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.156158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.156032    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tmp-dir\") pod \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\" (UID: \"bbbf5fe7-9177-4bba-9c37-dafa009f2969\") "
Sep 17 23:35:50.156158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.156141    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:50.156364 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.156223    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbbf5fe7-9177-4bba-9c37-dafa009f2969-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:35:50.156484 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.156436    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bbbf5fe7-9177-4bba-9c37-dafa009f2969-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.156484 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.156460    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.156662 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.156640    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-home" (OuterVolumeSpecName: "home") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:50.158752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.158731    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kube-api-access-8tz9k" (OuterVolumeSpecName: "kube-api-access-8tz9k") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "kube-api-access-8tz9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:35:50.158752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.158734    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:35:50.158919 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.158734    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-dshm" (OuterVolumeSpecName: "dshm") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:50.168634 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.168598    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:50.186784 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.186726    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbbf5fe7-9177-4bba-9c37-dafa009f2969" (UID: "bbbf5fe7-9177-4bba-9c37-dafa009f2969"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:35:50.257643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.257556    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.257643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.257585    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.257643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.257593    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.257643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.257602    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.257643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.257631    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbbf5fe7-9177-4bba-9c37-dafa009f2969-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.257643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.257643    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tz9k\" (UniqueName: \"kubernetes.io/projected/bbbf5fe7-9177-4bba-9c37-dafa009f2969-kube-api-access-8tz9k\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:35:50.884159 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.884121    2562 generic.go:356] "Generic (PLEG): container finished" podID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerID="3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175" exitCode=0
Sep 17 23:35:50.884659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.884181    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"
Sep 17 23:35:50.884659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.884207    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" event={"ID":"bbbf5fe7-9177-4bba-9c37-dafa009f2969","Type":"ContainerDied","Data":"3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175"}
Sep 17 23:35:50.884659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.884264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh" event={"ID":"bbbf5fe7-9177-4bba-9c37-dafa009f2969","Type":"ContainerDied","Data":"6a7448f02cfdd830b6d06d2b2d606fe1acbb7af525bc68081d6c10b859c4e6ad"}
Sep 17 23:35:50.884659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.884285    2562 scope.go:117] "RemoveContainer" containerID="3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175"
Sep 17 23:35:50.893678 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.893659    2562 scope.go:117] "RemoveContainer" containerID="4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683"
Sep 17 23:35:50.902274 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.902249    2562 scope.go:117] "RemoveContainer" containerID="3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175"
Sep 17 23:35:50.902640 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:35:50.902622    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175\": container with ID starting with 3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175 not found: ID does not exist" containerID="3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175"
Sep 17 23:35:50.902687 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.902651    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175"} err="failed to get container status \"3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175\": rpc error: code = NotFound desc = could not find container \"3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175\": container with ID starting with 3adb486f648c3b5851b7e858a16190fab8b3d9930c171309b5cea1986fc09175 not found: ID does not exist"
Sep 17 23:35:50.902687 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.902670    2562 scope.go:117] "RemoveContainer" containerID="4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683"
Sep 17 23:35:50.902969 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:35:50.902949    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683\": container with ID starting with 4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683 not found: ID does not exist" containerID="4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683"
Sep 17 23:35:50.903020 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.902976    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683"} err="failed to get container status \"4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683\": rpc error: code = NotFound desc = could not find container \"4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683\": container with ID starting with 4db3e172df1ff1c0d16d6b91b0aeeccc0ef00741884bb235a3385ea462834683 not found: ID does not exist"
Sep 17 23:35:50.908412 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.908375    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"]
Sep 17 23:35:50.913219 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:50.913184    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-6899f4fc85-knkwh"]
Sep 17 23:35:51.419760 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:35:51.419732    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" path="/var/lib/kubelet/pods/bbbf5fe7-9177-4bba-9c37-dafa009f2969/volumes"
Sep 17 23:36:23.326783 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.326744    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"]
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327098    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327110    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327119    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" containerName="main"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327125    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" containerName="main"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327139    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="storage-initializer"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327145    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="storage-initializer"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327154    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" containerName="storage-initializer"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327159    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" containerName="storage-initializer"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327217    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="34a98844-0963-45aa-8c9e-073bb98c592d" containerName="main"
Sep 17 23:36:23.327290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.327227    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbbf5fe7-9177-4bba-9c37-dafa009f2969" containerName="main"
Sep 17 23:36:23.330340 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.330323    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.333336 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.333309    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Sep 17 23:36:23.333783 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.333768    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 17 23:36:23.334136 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.334119    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:36:23.334631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.334607    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 17 23:36:23.334717 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.334630    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 17 23:36:23.334717 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.334695    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-cz6v9\""
Sep 17 23:36:23.342211 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.342186    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"]
Sep 17 23:36:23.345296 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345274    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.345399 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345309    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-model-cache\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.345399 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-dshm\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.345399 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345372    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.345532 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf295a0-932b-4203-a254-7b61c95d9f7c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.345532 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345452    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-home\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.345532 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345468    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf295a0-932b-4203-a254-7b61c95d9f7c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.345532 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.345482    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55lq\" (UniqueName: \"kubernetes.io/projected/ebf295a0-932b-4203-a254-7b61c95d9f7c-kube-api-access-l55lq\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.445916 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.445864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.445916 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.445918    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-model-cache\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.445952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-dshm\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.445972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.445992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf295a0-932b-4203-a254-7b61c95d9f7c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446031    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-home\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446054    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf295a0-932b-4203-a254-7b61c95d9f7c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446079    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55lq\" (UniqueName: \"kubernetes.io/projected/ebf295a0-932b-4203-a254-7b61c95d9f7c-kube-api-access-l55lq\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446329    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-tmp-dir\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446451    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-home\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446666 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446516    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-model-cache\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.446882 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.446863    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf295a0-932b-4203-a254-7b61c95d9f7c-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.448393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.448366    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-dshm\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.448508 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.448490    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf295a0-932b-4203-a254-7b61c95d9f7c-tls-certs\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.454004 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.453981    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55lq\" (UniqueName: \"kubernetes.io/projected/ebf295a0-932b-4203-a254-7b61c95d9f7c-kube-api-access-l55lq\") pod \"tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.641166 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.641119    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:23.918536 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:23.918511    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"]
Sep 17 23:36:23.920816 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:36:23.920785    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf295a0_932b_4203_a254_7b61c95d9f7c.slice/crio-0bbc9df80865d2f65935d4eb1e5f911565688a2aad0d49ff00f5d10f89829608 WatchSource:0}: Error finding container 0bbc9df80865d2f65935d4eb1e5f911565688a2aad0d49ff00f5d10f89829608: Status 404 returned error can't find the container with id 0bbc9df80865d2f65935d4eb1e5f911565688a2aad0d49ff00f5d10f89829608
Sep 17 23:36:24.054775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:24.054732    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" event={"ID":"ebf295a0-932b-4203-a254-7b61c95d9f7c","Type":"ContainerStarted","Data":"e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4"}
Sep 17 23:36:24.054955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:24.054806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" event={"ID":"ebf295a0-932b-4203-a254-7b61c95d9f7c","Type":"ContainerStarted","Data":"0bbc9df80865d2f65935d4eb1e5f911565688a2aad0d49ff00f5d10f89829608"}
Sep 17 23:36:27.077209 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:27.077170    2562 generic.go:356] "Generic (PLEG): container finished" podID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerID="e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4" exitCode=0
Sep 17 23:36:27.077209 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:27.077212    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" event={"ID":"ebf295a0-932b-4203-a254-7b61c95d9f7c","Type":"ContainerDied","Data":"e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4"}
Sep 17 23:36:28.084050 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:28.084010    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" event={"ID":"ebf295a0-932b-4203-a254-7b61c95d9f7c","Type":"ContainerStarted","Data":"2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba"}
Sep 17 23:36:28.105550 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:28.105493    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" podStartSLOduration=2.838821839 podStartE2EDuration="5.105475492s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.266653653s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:36:28.103497729 +0000 UTC m=+1549.198024780" watchObservedRunningTime="2026-09-17 23:36:28.105475492 +0000 UTC m=+1549.200002528"
Sep 17 23:36:28.641749 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:28.641705    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:33.641912 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:33.641859    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:33.654261 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:33.654233    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:34.124298 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:34.124266    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:36:58.510637 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.510604    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"]
Sep 17 23:36:58.513679 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.513660    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.516332 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.516308    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 17 23:36:58.516484 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.516309    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:36:58.516484 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.516361    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 17 23:36:58.516484 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.516471    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Sep 17 23:36:58.517419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.517394    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 17 23:36:58.517596 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.517399    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-ctv8h\""
Sep 17 23:36:58.524873 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.524850    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"]
Sep 17 23:36:58.665216 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665175    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tls-certs\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.665216 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kserve-provision-location\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.665453 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665244    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24f7j\" (UniqueName: \"kubernetes.io/projected/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kube-api-access-24f7j\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.665453 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665284    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a77c1ad6-006f-4701-bdf6-f2497a5accf9-cabundle-cert\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.665453 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-dshm\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.665453 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tmp-dir\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.665453 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-model-cache\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.665453 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.665373    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-home\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tls-certs\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766504    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kserve-provision-location\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766556 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766541    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24f7j\" (UniqueName: \"kubernetes.io/projected/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kube-api-access-24f7j\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766573    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a77c1ad6-006f-4701-bdf6-f2497a5accf9-cabundle-cert\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766602    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-dshm\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tmp-dir\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-model-cache\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.766865 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.766707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-home\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.767178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.767140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-home\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.767250 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.767222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-model-cache\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.767458 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.767427    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a77c1ad6-006f-4701-bdf6-f2497a5accf9-cabundle-cert\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.767458 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.767447    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kserve-provision-location\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.767586 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.767486    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tmp-dir\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.769129 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.769108    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-dshm\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.769278 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.769259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tls-certs\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.774441 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.774417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24f7j\" (UniqueName: \"kubernetes.io/projected/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kube-api-access-24f7j\") pod \"auth-disabled-test-kserve-5659bdb49d-r86tw\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:58.825347 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:58.825313    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:36:59.101169 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:59.101139    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"]
Sep 17 23:36:59.102089 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:36:59.102056    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77c1ad6_006f_4701_bdf6_f2497a5accf9.slice/crio-b0ec4518ee6449e44b728101a99e111ccb78f3e0feb401238eec4b3f1d383017 WatchSource:0}: Error finding container b0ec4518ee6449e44b728101a99e111ccb78f3e0feb401238eec4b3f1d383017: Status 404 returned error can't find the container with id b0ec4518ee6449e44b728101a99e111ccb78f3e0feb401238eec4b3f1d383017
Sep 17 23:36:59.241376 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:59.241339    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" event={"ID":"a77c1ad6-006f-4701-bdf6-f2497a5accf9","Type":"ContainerStarted","Data":"ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1"}
Sep 17 23:36:59.241521 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:36:59.241408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" event={"ID":"a77c1ad6-006f-4701-bdf6-f2497a5accf9","Type":"ContainerStarted","Data":"b0ec4518ee6449e44b728101a99e111ccb78f3e0feb401238eec4b3f1d383017"}
Sep 17 23:37:02.262738 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:02.262705    2562 generic.go:356] "Generic (PLEG): container finished" podID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerID="ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1" exitCode=0
Sep 17 23:37:02.263185 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:02.262782    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" event={"ID":"a77c1ad6-006f-4701-bdf6-f2497a5accf9","Type":"ContainerDied","Data":"ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1"}
Sep 17 23:37:03.270312 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:03.270275    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" event={"ID":"a77c1ad6-006f-4701-bdf6-f2497a5accf9","Type":"ContainerStarted","Data":"6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7"}
Sep 17 23:37:03.292337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:03.292283    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podStartSLOduration=3.067055962 podStartE2EDuration="5.292263744s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.225207782s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:36:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:37:03.291435792 +0000 UTC m=+1584.385962844" watchObservedRunningTime="2026-09-17 23:37:03.292263744 +0000 UTC m=+1584.386790779"
Sep 17 23:37:03.826408 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:03.826363    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:37:08.826156 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:08.826129    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:37:08.827790 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:08.827757    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:37:18.825965 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:18.825893    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:37:22.260145 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.260110    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"]
Sep 17 23:37:22.269519 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:37:22.269475    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:37:22.269519 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:37:22.269519 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:37:22.269519 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba" cmd=["/bin/sleep","15"]
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.269544    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" message="" err=<
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:37:22.269562    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerName="main" containerID="cri-o://2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba"
Sep 17 23:37:22.269860 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.269617    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerName="main" containerID="cri-o://2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba" gracePeriod=60
Sep 17 23:37:22.625757 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.625732    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:37:22.661733 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661689    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf295a0-932b-4203-a254-7b61c95d9f7c-cabundle-cert\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.661733 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661728    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l55lq\" (UniqueName: \"kubernetes.io/projected/ebf295a0-932b-4203-a254-7b61c95d9f7c-kube-api-access-l55lq\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.662052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661769    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf295a0-932b-4203-a254-7b61c95d9f7c-tls-certs\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.662052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661808    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-home\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.662052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661854    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-dshm\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.662052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661875    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-model-cache\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.662052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661915    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-kserve-provision-location\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.662052 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.661936    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-tmp-dir\") pod \"ebf295a0-932b-4203-a254-7b61c95d9f7c\" (UID: \"ebf295a0-932b-4203-a254-7b61c95d9f7c\") "
Sep 17 23:37:22.662362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.662116    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebf295a0-932b-4203-a254-7b61c95d9f7c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:37:22.662362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.662115    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-home" (OuterVolumeSpecName: "home") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.662362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.662208    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.662362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.662225    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ebf295a0-932b-4203-a254-7b61c95d9f7c-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.662362 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.662303    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.662593 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.662427    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.664678 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.664648    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebf295a0-932b-4203-a254-7b61c95d9f7c-kube-api-access-l55lq" (OuterVolumeSpecName: "kube-api-access-l55lq") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "kube-api-access-l55lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:37:22.664801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.664693    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-dshm" (OuterVolumeSpecName: "dshm") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.664801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.664693    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebf295a0-932b-4203-a254-7b61c95d9f7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:37:22.691481 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.691448    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ebf295a0-932b-4203-a254-7b61c95d9f7c" (UID: "ebf295a0-932b-4203-a254-7b61c95d9f7c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:37:22.763416 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.763382    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55lq\" (UniqueName: \"kubernetes.io/projected/ebf295a0-932b-4203-a254-7b61c95d9f7c-kube-api-access-l55lq\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.763416 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.763414    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebf295a0-932b-4203-a254-7b61c95d9f7c-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.763416 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.763422    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.763632 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.763431    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.763632 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.763438    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:22.763632 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:22.763446    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebf295a0-932b-4203-a254-7b61c95d9f7c-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:37:23.391659 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.391618    2562 generic.go:356] "Generic (PLEG): container finished" podID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerID="2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba" exitCode=0
Sep 17 23:37:23.392088 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.391664    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"
Sep 17 23:37:23.392088 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.391696    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" event={"ID":"ebf295a0-932b-4203-a254-7b61c95d9f7c","Type":"ContainerDied","Data":"2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba"}
Sep 17 23:37:23.392088 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.391752    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7" event={"ID":"ebf295a0-932b-4203-a254-7b61c95d9f7c","Type":"ContainerDied","Data":"0bbc9df80865d2f65935d4eb1e5f911565688a2aad0d49ff00f5d10f89829608"}
Sep 17 23:37:23.392088 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.391775    2562 scope.go:117] "RemoveContainer" containerID="2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba"
Sep 17 23:37:23.402235 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.402218    2562 scope.go:117] "RemoveContainer" containerID="e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4"
Sep 17 23:37:23.412322 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.412302    2562 scope.go:117] "RemoveContainer" containerID="2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba"
Sep 17 23:37:23.412661 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:37:23.412630    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba\": container with ID starting with 2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba not found: ID does not exist" containerID="2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba"
Sep 17 23:37:23.412752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.412674    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba"} err="failed to get container status \"2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba\": rpc error: code = NotFound desc = could not find container \"2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba\": container with ID starting with 2951e98aa2f41df094bf361356e3ed3cdc0b6235015141f575aa0542c2e488ba not found: ID does not exist"
Sep 17 23:37:23.412752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.412702    2562 scope.go:117] "RemoveContainer" containerID="e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4"
Sep 17 23:37:23.413256 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:37:23.413220    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4\": container with ID starting with e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4 not found: ID does not exist" containerID="e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4"
Sep 17 23:37:23.413366 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.413256    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4"} err="failed to get container status \"e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4\": rpc error: code = NotFound desc = could not find container \"e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4\": container with ID starting with e2c1a372b8d81928f83cdf0c1ad92bbbd55f0fc9c47bf9c5c440259198c232c4 not found: ID does not exist"
Sep 17 23:37:23.419549 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.419523    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"]
Sep 17 23:37:23.419641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:23.419562    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-5b68f7f5b-ch6n7"]
Sep 17 23:37:25.419646 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:25.419616    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" path="/var/lib/kubelet/pods/ebf295a0-932b-4203-a254-7b61c95d9f7c/volumes"
Sep 17 23:37:28.826568 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:28.826516    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:37:38.826715 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:38.826614    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:37:48.826577 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:48.826515    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:37:58.826155 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:37:58.826106    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:38:08.826435 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:38:08.826392    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:38:18.826403 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:38:18.826359    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:38:28.826086 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:38:28.826037    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:38:38.826635 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:38:38.826587    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:38:48.826187 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:38:48.826135    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 17 23:38:58.835790 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:38:58.835747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:38:58.843925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:38:58.843869    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:39:03.049212 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:03.049014    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"]
Sep 17 23:39:18.074932 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.074856    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main" containerID="cri-o://6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7" gracePeriod=45
Sep 17 23:39:18.827673 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.827641    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:39:18.900050 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.899959    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-home\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900050 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900003    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-dshm\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900050 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900026    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a77c1ad6-006f-4701-bdf6-f2497a5accf9-cabundle-cert\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900078    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tmp-dir\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900102    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tls-certs\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900135    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24f7j\" (UniqueName: \"kubernetes.io/projected/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kube-api-access-24f7j\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900166    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-model-cache\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900200    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kserve-provision-location\") pod \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\" (UID: \"a77c1ad6-006f-4701-bdf6-f2497a5accf9\") "
Sep 17 23:39:18.900617 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900529    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-model-cache" (OuterVolumeSpecName: "model-cache") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:18.900752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900732    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-home" (OuterVolumeSpecName: "home") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:18.900818 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.900539    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77c1ad6-006f-4701-bdf6-f2497a5accf9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:39:18.902427 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.902398    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-dshm" (OuterVolumeSpecName: "dshm") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:18.902583 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.902554    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:39:18.902651 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.902575    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kube-api-access-24f7j" (OuterVolumeSpecName: "kube-api-access-24f7j") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "kube-api-access-24f7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:39:18.912566 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.912523    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:18.923801 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:18.923767    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a77c1ad6-006f-4701-bdf6-f2497a5accf9" (UID: "a77c1ad6-006f-4701-bdf6-f2497a5accf9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:39:19.001642 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001602    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a77c1ad6-006f-4701-bdf6-f2497a5accf9-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.001642 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001634    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.001642 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001642    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77c1ad6-006f-4701-bdf6-f2497a5accf9-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.001642 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001651    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24f7j\" (UniqueName: \"kubernetes.io/projected/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kube-api-access-24f7j\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.001889 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001661    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.001889 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001671    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.001889 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001679    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.001889 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.001690    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77c1ad6-006f-4701-bdf6-f2497a5accf9-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:39:19.037776 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.037737    2562 generic.go:356] "Generic (PLEG): container finished" podID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerID="6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7" exitCode=0
Sep 17 23:39:19.037965 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.037795    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"
Sep 17 23:39:19.037965 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.037809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" event={"ID":"a77c1ad6-006f-4701-bdf6-f2497a5accf9","Type":"ContainerDied","Data":"6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7"}
Sep 17 23:39:19.037965 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.037872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw" event={"ID":"a77c1ad6-006f-4701-bdf6-f2497a5accf9","Type":"ContainerDied","Data":"b0ec4518ee6449e44b728101a99e111ccb78f3e0feb401238eec4b3f1d383017"}
Sep 17 23:39:19.037965 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.037921    2562 scope.go:117] "RemoveContainer" containerID="6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7"
Sep 17 23:39:19.047619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.047601    2562 scope.go:117] "RemoveContainer" containerID="ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1"
Sep 17 23:39:19.055978 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.055957    2562 scope.go:117] "RemoveContainer" containerID="6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7"
Sep 17 23:39:19.056260 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:39:19.056241    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7\": container with ID starting with 6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7 not found: ID does not exist" containerID="6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7"
Sep 17 23:39:19.056335 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.056265    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7"} err="failed to get container status \"6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7\": rpc error: code = NotFound desc = could not find container \"6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7\": container with ID starting with 6a2c72dba8da85e22a1678a3e1da1f161105dc024f8a2c50393633f9e6c47df7 not found: ID does not exist"
Sep 17 23:39:19.056335 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.056280    2562 scope.go:117] "RemoveContainer" containerID="ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1"
Sep 17 23:39:19.056518 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:39:19.056504    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1\": container with ID starting with ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1 not found: ID does not exist" containerID="ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1"
Sep 17 23:39:19.056559 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.056521    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1"} err="failed to get container status \"ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1\": rpc error: code = NotFound desc = could not find container \"ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1\": container with ID starting with ad95bac041ff9cf032d4b8cf4eb5f5fe848ded9a4679f8f750fcd9b2023bdcd1 not found: ID does not exist"
Sep 17 23:39:19.062059 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.062026    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"]
Sep 17 23:39:19.065101 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.065079    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5659bdb49d-r86tw"]
Sep 17 23:39:19.418645 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:39:19.418609    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" path="/var/lib/kubelet/pods/a77c1ad6-006f-4701-bdf6-f2497a5accf9/volumes"
Sep 17 23:40:23.293619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.293585    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"]
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294012    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="storage-initializer"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294025    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="storage-initializer"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294036    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerName="main"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294043    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerName="main"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294063    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerName="storage-initializer"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294073    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerName="storage-initializer"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294083    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294088    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294150    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a77c1ad6-006f-4701-bdf6-f2497a5accf9" containerName="main"
Sep 17 23:40:23.294158 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.294159    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebf295a0-932b-4203-a254-7b61c95d9f7c" containerName="main"
Sep 17 23:40:23.297228 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.297209    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.299847 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.299819    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 17 23:40:23.301171 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.301153    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 17 23:40:23.301291 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.301208    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 17 23:40:23.301367 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.301294    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-2nw9w\""
Sep 17 23:40:23.308959 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.308937    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"]
Sep 17 23:40:23.457142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.457108    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.457349 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.457159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.457349 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.457219    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.457349 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.457253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.457349 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.457281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jjn\" (UniqueName: \"kubernetes.io/projected/c80cb867-9425-4638-9819-ef835f8158b2-kube-api-access-f5jjn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.457349 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.457326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80cb867-9425-4638-9819-ef835f8158b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.558775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.558691    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80cb867-9425-4638-9819-ef835f8158b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.558775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.558746    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.558775 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.558774    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.559240 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.558826    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.559240 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.558856    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.559240 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.558896    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jjn\" (UniqueName: \"kubernetes.io/projected/c80cb867-9425-4638-9819-ef835f8158b2-kube-api-access-f5jjn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.559358 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.559280    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.559399 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.559376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.559439 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.559417    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.561107 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.561086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.561651 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.561629    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80cb867-9425-4638-9819-ef835f8158b2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.567190 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.567168    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jjn\" (UniqueName: \"kubernetes.io/projected/c80cb867-9425-4638-9819-ef835f8158b2-kube-api-access-f5jjn\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.608034 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.608011    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:23.877773 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:23.877746    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"]
Sep 17 23:40:23.879629 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:40:23.879599    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80cb867_9425_4638_9819_ef835f8158b2.slice/crio-54be1c7127a57564f011a9a850db9c878db21790f407119870cf145865c6d7f6 WatchSource:0}: Error finding container 54be1c7127a57564f011a9a850db9c878db21790f407119870cf145865c6d7f6: Status 404 returned error can't find the container with id 54be1c7127a57564f011a9a850db9c878db21790f407119870cf145865c6d7f6
Sep 17 23:40:24.378316 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:24.378277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" event={"ID":"c80cb867-9425-4638-9819-ef835f8158b2","Type":"ContainerStarted","Data":"1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3"}
Sep 17 23:40:24.378785 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:24.378337    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" event={"ID":"c80cb867-9425-4638-9819-ef835f8158b2","Type":"ContainerStarted","Data":"54be1c7127a57564f011a9a850db9c878db21790f407119870cf145865c6d7f6"}
Sep 17 23:40:24.396951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:24.396887    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" podStartSLOduration=1.396872353 podStartE2EDuration="1.396872353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:40:24.395444477 +0000 UTC m=+1785.489971523" watchObservedRunningTime="2026-09-17 23:40:24.396872353 +0000 UTC m=+1785.491399389"
Sep 17 23:40:24.608945 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:24.608895    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:33.609140 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:33.609104    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:33.621925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:33.621880    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:40:34.441582 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:40:34.441548    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:41:20.897809 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:20.897771    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"]
Sep 17 23:41:20.901244 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:20.901226    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:20.904312 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:20.904289    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 17 23:41:20.905504 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:20.905482    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 17 23:41:20.905619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:20.905482    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 17 23:41:20.905619 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:20.905515    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-nb7xx\""
Sep 17 23:41:20.915931 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:20.915892    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"]
Sep 17 23:41:21.062058 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.062023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.062241 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.062087    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.062241 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.062151    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.062241 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.062180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp52w\" (UniqueName: \"kubernetes.io/projected/75e84195-1e8c-4aab-a681-b1366a83210e-kube-api-access-fp52w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.062241 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.062218    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.062393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.062261    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e84195-1e8c-4aab-a681-b1366a83210e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.163628 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.163531    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e84195-1e8c-4aab-a681-b1366a83210e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.163796 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.163633    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.163796 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.163668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.163796 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.163775    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.163944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.163820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp52w\" (UniqueName: \"kubernetes.io/projected/75e84195-1e8c-4aab-a681-b1366a83210e-kube-api-access-fp52w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.163944 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.163873    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.164204 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.164175    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.164204 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.164193    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.164379 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.164259    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.166040 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.166019    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.166211 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.166195    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e84195-1e8c-4aab-a681-b1366a83210e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.172047 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.172025    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp52w\" (UniqueName: \"kubernetes.io/projected/75e84195-1e8c-4aab-a681-b1366a83210e-kube-api-access-fp52w\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.214989 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.214947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:21.487109 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.487081    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"]
Sep 17 23:41:21.488660 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:41:21.488623    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e84195_1e8c_4aab_a681_b1366a83210e.slice/crio-cea7b241039ecd962838dc806302b6afe5e37aa88e39eb036c10045ee2eaa119 WatchSource:0}: Error finding container cea7b241039ecd962838dc806302b6afe5e37aa88e39eb036c10045ee2eaa119: Status 404 returned error can't find the container with id cea7b241039ecd962838dc806302b6afe5e37aa88e39eb036c10045ee2eaa119
Sep 17 23:41:21.674012 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.673975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" event={"ID":"75e84195-1e8c-4aab-a681-b1366a83210e","Type":"ContainerStarted","Data":"ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b"}
Sep 17 23:41:21.674187 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.674027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" event={"ID":"75e84195-1e8c-4aab-a681-b1366a83210e","Type":"ContainerStarted","Data":"cea7b241039ecd962838dc806302b6afe5e37aa88e39eb036c10045ee2eaa119"}
Sep 17 23:41:21.693094 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:21.693037    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" podStartSLOduration=1.693018417 podStartE2EDuration="1.693018417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:41:21.690607481 +0000 UTC m=+1842.785134556" watchObservedRunningTime="2026-09-17 23:41:21.693018417 +0000 UTC m=+1842.787545453"
Sep 17 23:41:22.215783 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:22.215743    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:31.215533 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:31.215489    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:31.228217 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:31.228188    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:31.743868 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:31.743830    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:41:32.877724 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:32.877682    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"]
Sep 17 23:41:32.886839 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:41:32.886801    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:41:32.886839 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:41:32.886839 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:41:32.886839 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3" cmd=["/bin/sleep","15"]
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:32.886858    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" message="" err=<
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:41:32.886872    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" podUID="c80cb867-9425-4638-9819-ef835f8158b2" containerName="main" containerID="cri-o://1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3"
Sep 17 23:41:32.887110 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:32.886937    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" podUID="c80cb867-9425-4638-9819-ef835f8158b2" containerName="main" containerID="cri-o://1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3" gracePeriod=60
Sep 17 23:41:33.237216 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.237188    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:41:33.257319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257296    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-home\") pod \"c80cb867-9425-4638-9819-ef835f8158b2\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") "
Sep 17 23:41:33.257469 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257332    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-dshm\") pod \"c80cb867-9425-4638-9819-ef835f8158b2\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") "
Sep 17 23:41:33.257469 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257352    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5jjn\" (UniqueName: \"kubernetes.io/projected/c80cb867-9425-4638-9819-ef835f8158b2-kube-api-access-f5jjn\") pod \"c80cb867-9425-4638-9819-ef835f8158b2\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") "
Sep 17 23:41:33.257469 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257377    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-model-cache\") pod \"c80cb867-9425-4638-9819-ef835f8158b2\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") "
Sep 17 23:41:33.257469 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257452    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80cb867-9425-4638-9819-ef835f8158b2-tls-certs\") pod \"c80cb867-9425-4638-9819-ef835f8158b2\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") "
Sep 17 23:41:33.257697 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257497    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-tmp-dir\") pod \"c80cb867-9425-4638-9819-ef835f8158b2\" (UID: \"c80cb867-9425-4638-9819-ef835f8158b2\") "
Sep 17 23:41:33.257697 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257615    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-home" (OuterVolumeSpecName: "home") pod "c80cb867-9425-4638-9819-ef835f8158b2" (UID: "c80cb867-9425-4638-9819-ef835f8158b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:41:33.257814 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257799    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c80cb867-9425-4638-9819-ef835f8158b2" (UID: "c80cb867-9425-4638-9819-ef835f8158b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:41:33.257892 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257872    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:41:33.257978 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.257936    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c80cb867-9425-4638-9819-ef835f8158b2" (UID: "c80cb867-9425-4638-9819-ef835f8158b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:41:33.259871 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.259837    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80cb867-9425-4638-9819-ef835f8158b2-kube-api-access-f5jjn" (OuterVolumeSpecName: "kube-api-access-f5jjn") pod "c80cb867-9425-4638-9819-ef835f8158b2" (UID: "c80cb867-9425-4638-9819-ef835f8158b2"). InnerVolumeSpecName "kube-api-access-f5jjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:41:33.260643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.260622    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-dshm" (OuterVolumeSpecName: "dshm") pod "c80cb867-9425-4638-9819-ef835f8158b2" (UID: "c80cb867-9425-4638-9819-ef835f8158b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:41:33.260643 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.260630    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c80cb867-9425-4638-9819-ef835f8158b2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c80cb867-9425-4638-9819-ef835f8158b2" (UID: "c80cb867-9425-4638-9819-ef835f8158b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:41:33.359000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.358963    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:41:33.359000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.358994    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:41:33.359000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.359006    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5jjn\" (UniqueName: \"kubernetes.io/projected/c80cb867-9425-4638-9819-ef835f8158b2-kube-api-access-f5jjn\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:41:33.359244 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.359015    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c80cb867-9425-4638-9819-ef835f8158b2-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:41:33.359244 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.359026    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c80cb867-9425-4638-9819-ef835f8158b2-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:41:33.745482 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.745393    2562 generic.go:356] "Generic (PLEG): container finished" podID="c80cb867-9425-4638-9819-ef835f8158b2" containerID="1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3" exitCode=0
Sep 17 23:41:33.745482 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.745436    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"
Sep 17 23:41:33.745656 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.745440    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" event={"ID":"c80cb867-9425-4638-9819-ef835f8158b2","Type":"ContainerDied","Data":"1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3"}
Sep 17 23:41:33.745656 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.745571    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt" event={"ID":"c80cb867-9425-4638-9819-ef835f8158b2","Type":"ContainerDied","Data":"54be1c7127a57564f011a9a850db9c878db21790f407119870cf145865c6d7f6"}
Sep 17 23:41:33.745656 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.745603    2562 scope.go:117] "RemoveContainer" containerID="1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3"
Sep 17 23:41:33.754607 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.754591    2562 scope.go:117] "RemoveContainer" containerID="1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3"
Sep 17 23:41:33.754880 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:41:33.754862    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3\": container with ID starting with 1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3 not found: ID does not exist" containerID="1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3"
Sep 17 23:41:33.754964 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.754890    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3"} err="failed to get container status \"1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3\": rpc error: code = NotFound desc = could not find container \"1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3\": container with ID starting with 1d871a732fc2650fea4a9a04e0840303caf600e944de743567912ee88d8d7cb3 not found: ID does not exist"
Sep 17 23:41:33.761205 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.761181    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"]
Sep 17 23:41:33.765119 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:33.765099    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997vdktt"]
Sep 17 23:41:35.419209 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:35.419178    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c80cb867-9425-4638-9819-ef835f8158b2" path="/var/lib/kubelet/pods/c80cb867-9425-4638-9819-ef835f8158b2/volumes"
Sep 17 23:41:49.208849 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.208817    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"]
Sep 17 23:41:49.209354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.209187    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c80cb867-9425-4638-9819-ef835f8158b2" containerName="main"
Sep 17 23:41:49.209354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.209198    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c80cb867-9425-4638-9819-ef835f8158b2" containerName="main"
Sep 17 23:41:49.209354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.209257    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c80cb867-9425-4638-9819-ef835f8158b2" containerName="main"
Sep 17 23:41:49.212706 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.212688    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.215504 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.215479    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Sep 17 23:41:49.215980 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.215961    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 17 23:41:49.216125 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.216062    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 17 23:41:49.216311 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.216294    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:41:49.216380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.216351    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-4gwh7\""
Sep 17 23:41:49.216441 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.216376    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 17 23:41:49.224329 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.224308    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"]
Sep 17 23:41:49.296227 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296191    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.296227 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.296461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296247    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjnv5\" (UniqueName: \"kubernetes.io/projected/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kube-api-access-jjnv5\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.296461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8125408f-1f40-4b7f-9ab5-72f9557b3b81-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.296461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296325    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.296461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296349    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.296461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296376    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.296461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.296392    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397265 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397280    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397311    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397361    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397388    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397412    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjnv5\" (UniqueName: \"kubernetes.io/projected/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kube-api-access-jjnv5\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397461 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8125408f-1f40-4b7f-9ab5-72f9557b3b81-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397773 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397492    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397773 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397688    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397792    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397942 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397895    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.397994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.397951    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.398274 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.398256    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8125408f-1f40-4b7f-9ab5-72f9557b3b81-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.399668 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.399644    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.399843 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.399827    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.405118 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.405095    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjnv5\" (UniqueName: \"kubernetes.io/projected/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kube-api-access-jjnv5\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.524546 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.524443    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:49.803807 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.803781    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"]
Sep 17 23:41:49.805768 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:41:49.805737    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8125408f_1f40_4b7f_9ab5_72f9557b3b81.slice/crio-1a320c95c8ad2712dfba16eb0b862b60d51538c53cdf9c5872885a522bfbda50 WatchSource:0}: Error finding container 1a320c95c8ad2712dfba16eb0b862b60d51538c53cdf9c5872885a522bfbda50: Status 404 returned error can't find the container with id 1a320c95c8ad2712dfba16eb0b862b60d51538c53cdf9c5872885a522bfbda50
Sep 17 23:41:49.827383 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:49.827355    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" event={"ID":"8125408f-1f40-4b7f-9ab5-72f9557b3b81","Type":"ContainerStarted","Data":"1a320c95c8ad2712dfba16eb0b862b60d51538c53cdf9c5872885a522bfbda50"}
Sep 17 23:41:50.834113 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:50.834074    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" event={"ID":"8125408f-1f40-4b7f-9ab5-72f9557b3b81","Type":"ContainerStarted","Data":"b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced"}
Sep 17 23:41:52.845983 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:52.845946    2562 generic.go:356] "Generic (PLEG): container finished" podID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerID="b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced" exitCode=0
Sep 17 23:41:52.846466 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:52.846034    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" event={"ID":"8125408f-1f40-4b7f-9ab5-72f9557b3b81","Type":"ContainerDied","Data":"b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced"}
Sep 17 23:41:53.853441 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:53.853408    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" event={"ID":"8125408f-1f40-4b7f-9ab5-72f9557b3b81","Type":"ContainerStarted","Data":"bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6"}
Sep 17 23:41:53.877449 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:53.877390    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podStartSLOduration=2.6292041409999998 podStartE2EDuration="4.87737213s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.248167989s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:41:53.874226427 +0000 UTC m=+1874.968753487" watchObservedRunningTime="2026-09-17 23:41:53.87737213 +0000 UTC m=+1874.971899166"
Sep 17 23:41:54.524975 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:54.524941    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:59.525054 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:59.525019    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:41:59.526808 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:41:59.526783    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:42:04.317166 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.317089    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"]
Sep 17 23:42:04.327300 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:42:04.327264    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:42:04.327300 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:42:04.327300 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:42:04.327300 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b" cmd=["/bin/sleep","15"]
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.327323    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" message="" err=<
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:42:04.327342    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" podUID="75e84195-1e8c-4aab-a681-b1366a83210e" containerName="main" containerID="cri-o://ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b"
Sep 17 23:42:04.327590 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.327388    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" podUID="75e84195-1e8c-4aab-a681-b1366a83210e" containerName="main" containerID="cri-o://ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b" gracePeriod=60
Sep 17 23:42:04.678796 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.678758    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:42:04.732855 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.732827    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fp52w\" (UniqueName: \"kubernetes.io/projected/75e84195-1e8c-4aab-a681-b1366a83210e-kube-api-access-fp52w\") pod \"75e84195-1e8c-4aab-a681-b1366a83210e\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") "
Sep 17 23:42:04.733046 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.732880    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-home\") pod \"75e84195-1e8c-4aab-a681-b1366a83210e\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") "
Sep 17 23:42:04.733046 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.732919    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-dshm\") pod \"75e84195-1e8c-4aab-a681-b1366a83210e\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") "
Sep 17 23:42:04.733046 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.732971    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-model-cache\") pod \"75e84195-1e8c-4aab-a681-b1366a83210e\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") "
Sep 17 23:42:04.733046 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.733010    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-tmp-dir\") pod \"75e84195-1e8c-4aab-a681-b1366a83210e\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") "
Sep 17 23:42:04.733271 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.733062    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e84195-1e8c-4aab-a681-b1366a83210e-tls-certs\") pod \"75e84195-1e8c-4aab-a681-b1366a83210e\" (UID: \"75e84195-1e8c-4aab-a681-b1366a83210e\") "
Sep 17 23:42:04.733271 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.733166    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-home" (OuterVolumeSpecName: "home") pod "75e84195-1e8c-4aab-a681-b1366a83210e" (UID: "75e84195-1e8c-4aab-a681-b1366a83210e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:04.733271 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.733242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-model-cache" (OuterVolumeSpecName: "model-cache") pod "75e84195-1e8c-4aab-a681-b1366a83210e" (UID: "75e84195-1e8c-4aab-a681-b1366a83210e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:04.733423 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.733363    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:42:04.733423 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.733368    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75e84195-1e8c-4aab-a681-b1366a83210e" (UID: "75e84195-1e8c-4aab-a681-b1366a83210e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:04.733423 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.733382    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:42:04.735196 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.735164    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e84195-1e8c-4aab-a681-b1366a83210e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75e84195-1e8c-4aab-a681-b1366a83210e" (UID: "75e84195-1e8c-4aab-a681-b1366a83210e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:42:04.735196 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.735186    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e84195-1e8c-4aab-a681-b1366a83210e-kube-api-access-fp52w" (OuterVolumeSpecName: "kube-api-access-fp52w") pod "75e84195-1e8c-4aab-a681-b1366a83210e" (UID: "75e84195-1e8c-4aab-a681-b1366a83210e"). InnerVolumeSpecName "kube-api-access-fp52w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:42:04.735374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.735164    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-dshm" (OuterVolumeSpecName: "dshm") pod "75e84195-1e8c-4aab-a681-b1366a83210e" (UID: "75e84195-1e8c-4aab-a681-b1366a83210e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:42:04.834985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.834893    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:42:04.834985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.834984    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75e84195-1e8c-4aab-a681-b1366a83210e-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:42:04.835195 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.835000    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp52w\" (UniqueName: \"kubernetes.io/projected/75e84195-1e8c-4aab-a681-b1366a83210e-kube-api-access-fp52w\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:42:04.835195 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.835013    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75e84195-1e8c-4aab-a681-b1366a83210e-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:42:04.920683 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.920604    2562 generic.go:356] "Generic (PLEG): container finished" podID="75e84195-1e8c-4aab-a681-b1366a83210e" containerID="ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b" exitCode=0
Sep 17 23:42:04.920683 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.920651    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"
Sep 17 23:42:04.920857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.920686    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" event={"ID":"75e84195-1e8c-4aab-a681-b1366a83210e","Type":"ContainerDied","Data":"ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b"}
Sep 17 23:42:04.920857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.920775    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9" event={"ID":"75e84195-1e8c-4aab-a681-b1366a83210e","Type":"ContainerDied","Data":"cea7b241039ecd962838dc806302b6afe5e37aa88e39eb036c10045ee2eaa119"}
Sep 17 23:42:04.920857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.920803    2562 scope.go:117] "RemoveContainer" containerID="ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b"
Sep 17 23:42:04.932253 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.932232    2562 scope.go:117] "RemoveContainer" containerID="ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b"
Sep 17 23:42:04.932546 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:42:04.932527    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b\": container with ID starting with ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b not found: ID does not exist" containerID="ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b"
Sep 17 23:42:04.932607 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.932557    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b"} err="failed to get container status \"ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b\": rpc error: code = NotFound desc = could not find container \"ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b\": container with ID starting with ca46b10be5ce428c9664a1ad4da93c325f439bb5b703a05fcb796d26921eb78b not found: ID does not exist"
Sep 17 23:42:04.943317 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.943290    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"]
Sep 17 23:42:04.947163 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:04.947140    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995vwxq9"]
Sep 17 23:42:05.419508 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:05.419475    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e84195-1e8c-4aab-a681-b1366a83210e" path="/var/lib/kubelet/pods/75e84195-1e8c-4aab-a681-b1366a83210e/volumes"
Sep 17 23:42:09.525826 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:09.525779    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:42:19.525546 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:19.525500    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:42:29.525261 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:29.525214    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:42:39.525512 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:39.525464    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:42:49.525526 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:49.525476    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:42:59.524833 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:42:59.524791    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:43:09.525851 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:43:09.525800    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:43:19.525309 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:43:19.525259    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:43:29.524871 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:43:29.524823    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:43:39.525073 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:43:39.524977    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Sep 17 23:43:49.535187 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:43:49.535146    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:43:49.543270 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:43:49.543233    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:44:29.681637 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:29.681600    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"]
Sep 17 23:44:44.703925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:44.703866    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main" containerID="cri-o://bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6" gracePeriod=45
Sep 17 23:44:45.460993 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.460968    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:44:45.547970 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.547859    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-dshm\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.547970 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.547965    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjnv5\" (UniqueName: \"kubernetes.io/projected/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kube-api-access-jjnv5\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.548178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.547988    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-home\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.548178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548028    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-model-cache\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.548178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548063    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tmp-dir\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.548178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548103    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tls-certs\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.548178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548132    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8125408f-1f40-4b7f-9ab5-72f9557b3b81-cabundle-cert\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.548178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548164    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kserve-provision-location\") pod \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\" (UID: \"8125408f-1f40-4b7f-9ab5-72f9557b3b81\") "
Sep 17 23:44:45.548498 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548344    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-model-cache" (OuterVolumeSpecName: "model-cache") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:45.548564 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548501    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.548650 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.548626    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8125408f-1f40-4b7f-9ab5-72f9557b3b81-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:44:45.549054 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.549025    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-home" (OuterVolumeSpecName: "home") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:45.550391 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.550365    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-dshm" (OuterVolumeSpecName: "dshm") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:45.550483 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.550384    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kube-api-access-jjnv5" (OuterVolumeSpecName: "kube-api-access-jjnv5") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "kube-api-access-jjnv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:44:45.550483 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.550400    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:44:45.561480 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.561453    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:45.573033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.573010    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8125408f-1f40-4b7f-9ab5-72f9557b3b81" (UID: "8125408f-1f40-4b7f-9ab5-72f9557b3b81"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:44:45.649196 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.649160    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.649196 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.649194    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8125408f-1f40-4b7f-9ab5-72f9557b3b81-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.649196 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.649203    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8125408f-1f40-4b7f-9ab5-72f9557b3b81-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.649418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.649213    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.649418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.649222    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.649418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.649230    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjnv5\" (UniqueName: \"kubernetes.io/projected/8125408f-1f40-4b7f-9ab5-72f9557b3b81-kube-api-access-jjnv5\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.649418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.649238    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8125408f-1f40-4b7f-9ab5-72f9557b3b81-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:44:45.808066 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.807973    2562 generic.go:356] "Generic (PLEG): container finished" podID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerID="bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6" exitCode=0
Sep 17 23:44:45.808066 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.808020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" event={"ID":"8125408f-1f40-4b7f-9ab5-72f9557b3b81","Type":"ContainerDied","Data":"bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6"}
Sep 17 23:44:45.808573 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.808078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz" event={"ID":"8125408f-1f40-4b7f-9ab5-72f9557b3b81","Type":"ContainerDied","Data":"1a320c95c8ad2712dfba16eb0b862b60d51538c53cdf9c5872885a522bfbda50"}
Sep 17 23:44:45.808573 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.808105    2562 scope.go:117] "RemoveContainer" containerID="bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6"
Sep 17 23:44:45.808573 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.808031    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"
Sep 17 23:44:45.817698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.817682    2562 scope.go:117] "RemoveContainer" containerID="b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced"
Sep 17 23:44:45.825702 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.825682    2562 scope.go:117] "RemoveContainer" containerID="bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6"
Sep 17 23:44:45.825987 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:44:45.825958    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6\": container with ID starting with bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6 not found: ID does not exist" containerID="bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6"
Sep 17 23:44:45.826064 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.825996    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6"} err="failed to get container status \"bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6\": rpc error: code = NotFound desc = could not find container \"bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6\": container with ID starting with bc12d8daff803c2355e3ef0150f842a1a72f79e1b32bcdb8fdd356441eb7e2a6 not found: ID does not exist"
Sep 17 23:44:45.826064 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.826012    2562 scope.go:117] "RemoveContainer" containerID="b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced"
Sep 17 23:44:45.826250 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:44:45.826234    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced\": container with ID starting with b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced not found: ID does not exist" containerID="b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced"
Sep 17 23:44:45.826291 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.826253    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced"} err="failed to get container status \"b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced\": rpc error: code = NotFound desc = could not find container \"b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced\": container with ID starting with b43d945603749a981c1d89c6a5a1ee14e32e658ea859da9382c2ee1fe41d2ced not found: ID does not exist"
Sep 17 23:44:45.831026 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.831006    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"]
Sep 17 23:44:45.835286 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:45.835267    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6cdbd57775hv4kz"]
Sep 17 23:44:46.597348 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597316    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"]
Sep 17 23:44:46.597671 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597658    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main"
Sep 17 23:44:46.597720 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597674    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main"
Sep 17 23:44:46.597720 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597685    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="storage-initializer"
Sep 17 23:44:46.597720 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597691    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="storage-initializer"
Sep 17 23:44:46.597720 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597706    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e84195-1e8c-4aab-a681-b1366a83210e" containerName="main"
Sep 17 23:44:46.597720 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597713    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e84195-1e8c-4aab-a681-b1366a83210e" containerName="main"
Sep 17 23:44:46.597872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597771    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e84195-1e8c-4aab-a681-b1366a83210e" containerName="main"
Sep 17 23:44:46.597872 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.597783    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" containerName="main"
Sep 17 23:44:46.602272 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.602251    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.607091 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.607070    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 17 23:44:46.607220 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.607102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-6dldf\""
Sep 17 23:44:46.607220 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.607165    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 17 23:44:46.607413 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.607393    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 17 23:44:46.613734 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.613712    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"]
Sep 17 23:44:46.757798 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.757760    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.758033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.757809    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.758033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.757828    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.758033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.757881    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.758033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.757981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5m7h\" (UniqueName: \"kubernetes.io/projected/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-kube-api-access-w5m7h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.758033 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.758026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.858812 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.858722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.858812 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.858776    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.858812 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.858797    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.858812 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.858812    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.859419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.858850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5m7h\" (UniqueName: \"kubernetes.io/projected/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-kube-api-access-w5m7h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.859419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.858878    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.859419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.859291    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.859419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.859328    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.859419 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.859358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.861136 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.861113    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.861270 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.861246    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.867937 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.867886    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5m7h\" (UniqueName: \"kubernetes.io/projected/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-kube-api-access-w5m7h\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:46.913379 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:46.913340    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:47.191890 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:47.191854    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"]
Sep 17 23:44:47.193407 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:44:47.193380    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa4e84_50d2_4d25_b17a_91c5a73583a2.slice/crio-7fd6a48253b03139ec2e28ffc07dc01c89681f8a492507d5e07c1f79852b0a34 WatchSource:0}: Error finding container 7fd6a48253b03139ec2e28ffc07dc01c89681f8a492507d5e07c1f79852b0a34: Status 404 returned error can't find the container with id 7fd6a48253b03139ec2e28ffc07dc01c89681f8a492507d5e07c1f79852b0a34
Sep 17 23:44:47.419749 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:47.419714    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8125408f-1f40-4b7f-9ab5-72f9557b3b81" path="/var/lib/kubelet/pods/8125408f-1f40-4b7f-9ab5-72f9557b3b81/volumes"
Sep 17 23:44:47.824234 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:47.824188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" event={"ID":"a6aa4e84-50d2-4d25-b17a-91c5a73583a2","Type":"ContainerStarted","Data":"b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60"}
Sep 17 23:44:47.824418 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:47.824243    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" event={"ID":"a6aa4e84-50d2-4d25-b17a-91c5a73583a2","Type":"ContainerStarted","Data":"7fd6a48253b03139ec2e28ffc07dc01c89681f8a492507d5e07c1f79852b0a34"}
Sep 17 23:44:47.844203 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:47.844145    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" podStartSLOduration=1.844129701 podStartE2EDuration="1.844129701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:44:47.84272207 +0000 UTC m=+2048.937249106" watchObservedRunningTime="2026-09-17 23:44:47.844129701 +0000 UTC m=+2048.938656737"
Sep 17 23:44:47.914262 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:47.914210    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:56.914097 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:56.914033    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:56.926627 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:56.926599    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:44:57.887216 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:44:57.887187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:45:20.820891 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:20.820858    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"]
Sep 17 23:45:20.830658 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:45:20.830616    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 17 23:45:20.830658 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:45:20.830658 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:45:20.830658 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60" cmd=["/bin/sleep","15"]
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:20.830682    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" message="" err=<
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:45:20.830698    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" podUID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" containerName="main" containerID="cri-o://b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60"
Sep 17 23:45:20.830931 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:20.830753    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" podUID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" containerName="main" containerID="cri-o://b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60" gracePeriod=60
Sep 17 23:45:20.876396 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:20.876355    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" podUID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Sep 17 23:45:21.171028 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.171004    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:45:21.246804 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.246778    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tmp-dir\") pod \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") "
Sep 17 23:45:21.246985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.246846    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tls-certs\") pod \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") "
Sep 17 23:45:21.246985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.246876    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5m7h\" (UniqueName: \"kubernetes.io/projected/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-kube-api-access-w5m7h\") pod \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") "
Sep 17 23:45:21.246985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.246895    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-dshm\") pod \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") "
Sep 17 23:45:21.246985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.246945    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-model-cache\") pod \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") "
Sep 17 23:45:21.246985 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.246977    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-home\") pod \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\" (UID: \"a6aa4e84-50d2-4d25-b17a-91c5a73583a2\") "
Sep 17 23:45:21.247266 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.247062    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6aa4e84-50d2-4d25-b17a-91c5a73583a2" (UID: "a6aa4e84-50d2-4d25-b17a-91c5a73583a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:21.247266 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.247229    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6aa4e84-50d2-4d25-b17a-91c5a73583a2" (UID: "a6aa4e84-50d2-4d25-b17a-91c5a73583a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:21.247266 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.247255    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-home" (OuterVolumeSpecName: "home") pod "a6aa4e84-50d2-4d25-b17a-91c5a73583a2" (UID: "a6aa4e84-50d2-4d25-b17a-91c5a73583a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:21.247426 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.247341    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:45:21.247426 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.247359    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:45:21.247426 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.247372    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:45:21.249209 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.249179    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6aa4e84-50d2-4d25-b17a-91c5a73583a2" (UID: "a6aa4e84-50d2-4d25-b17a-91c5a73583a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:45:21.249319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.249235    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-kube-api-access-w5m7h" (OuterVolumeSpecName: "kube-api-access-w5m7h") pod "a6aa4e84-50d2-4d25-b17a-91c5a73583a2" (UID: "a6aa4e84-50d2-4d25-b17a-91c5a73583a2"). InnerVolumeSpecName "kube-api-access-w5m7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:45:21.249319 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.249262    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-dshm" (OuterVolumeSpecName: "dshm") pod "a6aa4e84-50d2-4d25-b17a-91c5a73583a2" (UID: "a6aa4e84-50d2-4d25-b17a-91c5a73583a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:45:21.348068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.347981    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:45:21.348068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.348012    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5m7h\" (UniqueName: \"kubernetes.io/projected/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-kube-api-access-w5m7h\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:45:21.348068 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:21.348022    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6aa4e84-50d2-4d25-b17a-91c5a73583a2-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:45:22.007806 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.007772    2562 generic.go:356] "Generic (PLEG): container finished" podID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" containerID="b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60" exitCode=0
Sep 17 23:45:22.008256 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.007816    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"
Sep 17 23:45:22.008256 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.007856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" event={"ID":"a6aa4e84-50d2-4d25-b17a-91c5a73583a2","Type":"ContainerDied","Data":"b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60"}
Sep 17 23:45:22.008256 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.007925    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg" event={"ID":"a6aa4e84-50d2-4d25-b17a-91c5a73583a2","Type":"ContainerDied","Data":"7fd6a48253b03139ec2e28ffc07dc01c89681f8a492507d5e07c1f79852b0a34"}
Sep 17 23:45:22.008256 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.007947    2562 scope.go:117] "RemoveContainer" containerID="b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60"
Sep 17 23:45:22.017282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.017258    2562 scope.go:117] "RemoveContainer" containerID="b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60"
Sep 17 23:45:22.017598 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:45:22.017575    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60\": container with ID starting with b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60 not found: ID does not exist" containerID="b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60"
Sep 17 23:45:22.017648 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.017607    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60"} err="failed to get container status \"b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60\": rpc error: code = NotFound desc = could not find container \"b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60\": container with ID starting with b4022b32cf3f7b7ac434710a94bd7af67d2a032d1e25cd119de138e004a5ff60 not found: ID does not exist"
Sep 17 23:45:22.024883 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.024851    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"]
Sep 17 23:45:22.026866 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:22.026844    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-27nhg"]
Sep 17 23:45:23.419082 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:23.419045    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" path="/var/lib/kubelet/pods/a6aa4e84-50d2-4d25-b17a-91c5a73583a2/volumes"
Sep 17 23:45:37.606475 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.606442    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"]
Sep 17 23:45:37.607088 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.606980    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" containerName="main"
Sep 17 23:45:37.607088 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.607000    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" containerName="main"
Sep 17 23:45:37.607226 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.607100    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6aa4e84-50d2-4d25-b17a-91c5a73583a2" containerName="main"
Sep 17 23:45:37.611734 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.611713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.614870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.614842    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-6lp8v\""
Sep 17 23:45:37.614870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.614853    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 17 23:45:37.615073 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.614916    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 17 23:45:37.615073 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.614968    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 17 23:45:37.619857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.619836    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"]
Sep 17 23:45:37.689468 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.689428    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.689657 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.689491    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b19bb5f-a003-4ea5-960c-0a1169f785db-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.689657 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.689584    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q5vw\" (UniqueName: \"kubernetes.io/projected/0b19bb5f-a003-4ea5-960c-0a1169f785db-kube-api-access-9q5vw\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.689756 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.689659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.689756 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.689691    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.689756 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.689711    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.689756 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.689729    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791119 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791073    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b19bb5f-a003-4ea5-960c-0a1169f785db-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791119 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791125    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q5vw\" (UniqueName: \"kubernetes.io/projected/0b19bb5f-a003-4ea5-960c-0a1169f785db-kube-api-access-9q5vw\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791169    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791202    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791353    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791547 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791403    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791547 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791445    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791752 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791844 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791751    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.791844 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791819    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.792005 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.791948    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.793445 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.793428    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.793722 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.793705    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b19bb5f-a003-4ea5-960c-0a1169f785db-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.798674 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.798653    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q5vw\" (UniqueName: \"kubernetes.io/projected/0b19bb5f-a003-4ea5-960c-0a1169f785db-kube-api-access-9q5vw\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:37.923507 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:37.923411    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:38.199166 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:38.199137    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"]
Sep 17 23:45:38.244336 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:38.244313    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 23:45:39.098817 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:39.098780    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerStarted","Data":"de6a54dfde49bdd8734ed02e2601eab87d8661f23da25fffbb69448c0cdcbf48"}
Sep 17 23:45:48.150337 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:48.150299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerStarted","Data":"c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c"}
Sep 17 23:45:49.157176 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:49.157134    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerID="c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c" exitCode=0
Sep 17 23:45:49.157565 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:49.157218    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerDied","Data":"c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c"}
Sep 17 23:45:50.165123 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:50.165082    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerStarted","Data":"b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da"}
Sep 17 23:45:50.165494 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:50.165154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerStarted","Data":"e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8"}
Sep 17 23:45:50.186426 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:50.186376    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podStartSLOduration=3.03840089 podStartE2EDuration="13.186357963s" totalImagesPullingTime="8.960849253s" totalInitContainerRuntime="1.18710782s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:45:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:45:50.183825219 +0000 UTC m=+2111.278352255" watchObservedRunningTime="2026-09-17 23:45:50.186357963 +0000 UTC m=+2111.280884999"
Sep 17 23:45:50.924268 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:50.924226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:57.924178 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:57.924129    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:45:57.925768 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:45:57.925742    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:46:07.923857 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:46:07.923813    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:46:17.924386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:46:17.924345    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:46:27.924350 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:46:27.924281    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:46:37.924307 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:46:37.924208    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:46:47.924338 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:46:47.924294    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:46:57.924118 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:46:57.924072    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:47:07.924527 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:07.924479    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:47:17.924631 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:17.924584    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:47:27.924531 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:27.924482    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:47:37.924734 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:37.924687    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:47:47.935205 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:47.935170    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:47:47.943792 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:47.943760    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:47:52.836672 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:52.836636    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"]
Sep 17 23:47:52.837152 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:52.836978    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="modelcar" containerID="cri-o://b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da" gracePeriod=60
Sep 17 23:47:53.856357 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:53.856325    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9_0b19bb5f-a003-4ea5-960c-0a1169f785db/modelcar/0.log"
Sep 17 23:47:53.856978 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:53.856941    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerID="b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da" exitCode=143
Sep 17 23:47:53.857087 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:47:53.857012    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerDied","Data":"b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da"}
Sep 17 23:48:07.861324 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:07.861273    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" containerID="cri-o://e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8" gracePeriod=45
Sep 17 23:48:07.936343 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:07.936294    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Sep 17 23:48:08.637988 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.637953    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9_0b19bb5f-a003-4ea5-960c-0a1169f785db/modelcar/0.log"
Sep 17 23:48:08.640844 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.640819    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:48:08.706325 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706298    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-model-cache\") pod \"0b19bb5f-a003-4ea5-960c-0a1169f785db\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") "
Sep 17 23:48:08.706509 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706367    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-home\") pod \"0b19bb5f-a003-4ea5-960c-0a1169f785db\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") "
Sep 17 23:48:08.706509 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706392    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b19bb5f-a003-4ea5-960c-0a1169f785db-tls-certs\") pod \"0b19bb5f-a003-4ea5-960c-0a1169f785db\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") "
Sep 17 23:48:08.706509 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706439    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q5vw\" (UniqueName: \"kubernetes.io/projected/0b19bb5f-a003-4ea5-960c-0a1169f785db-kube-api-access-9q5vw\") pod \"0b19bb5f-a003-4ea5-960c-0a1169f785db\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") "
Sep 17 23:48:08.706509 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706462    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-dshm\") pod \"0b19bb5f-a003-4ea5-960c-0a1169f785db\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") "
Sep 17 23:48:08.706509 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706484    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-tmp-dir\") pod \"0b19bb5f-a003-4ea5-960c-0a1169f785db\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") "
Sep 17 23:48:08.706771 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706524    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b19bb5f-a003-4ea5-960c-0a1169f785db" (UID: "0b19bb5f-a003-4ea5-960c-0a1169f785db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:48:08.706771 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706560    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-kserve-provision-location\") pod \"0b19bb5f-a003-4ea5-960c-0a1169f785db\" (UID: \"0b19bb5f-a003-4ea5-960c-0a1169f785db\") "
Sep 17 23:48:08.706882 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706797    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:48:08.706976 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.706947    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0b19bb5f-a003-4ea5-960c-0a1169f785db" (UID: "0b19bb5f-a003-4ea5-960c-0a1169f785db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:48:08.707221 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.707195    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-home" (OuterVolumeSpecName: "home") pod "0b19bb5f-a003-4ea5-960c-0a1169f785db" (UID: "0b19bb5f-a003-4ea5-960c-0a1169f785db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:48:08.708542 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.708512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-dshm" (OuterVolumeSpecName: "dshm") pod "0b19bb5f-a003-4ea5-960c-0a1169f785db" (UID: "0b19bb5f-a003-4ea5-960c-0a1169f785db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:48:08.708634 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.708593    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b19bb5f-a003-4ea5-960c-0a1169f785db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b19bb5f-a003-4ea5-960c-0a1169f785db" (UID: "0b19bb5f-a003-4ea5-960c-0a1169f785db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:48:08.708704 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.708681    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b19bb5f-a003-4ea5-960c-0a1169f785db-kube-api-access-9q5vw" (OuterVolumeSpecName: "kube-api-access-9q5vw") pod "0b19bb5f-a003-4ea5-960c-0a1169f785db" (UID: "0b19bb5f-a003-4ea5-960c-0a1169f785db"). InnerVolumeSpecName "kube-api-access-9q5vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:48:08.717610 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.717575    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b19bb5f-a003-4ea5-960c-0a1169f785db" (UID: "0b19bb5f-a003-4ea5-960c-0a1169f785db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:48:08.808142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.808107    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:48:08.808142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.808136    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b19bb5f-a003-4ea5-960c-0a1169f785db-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:48:08.808142 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.808146    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q5vw\" (UniqueName: \"kubernetes.io/projected/0b19bb5f-a003-4ea5-960c-0a1169f785db-kube-api-access-9q5vw\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:48:08.808374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.808162    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:48:08.808374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.808193    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:48:08.808374 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.808202    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b19bb5f-a003-4ea5-960c-0a1169f785db-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:48:08.935026 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.934951    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9_0b19bb5f-a003-4ea5-960c-0a1169f785db/modelcar/0.log"
Sep 17 23:48:08.935615 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.935590    2562 generic.go:356] "Generic (PLEG): container finished" podID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerID="e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8" exitCode=0
Sep 17 23:48:08.935691 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.935674    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"
Sep 17 23:48:08.935736 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.935679    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerDied","Data":"e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8"}
Sep 17 23:48:08.935778 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.935767    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9" event={"ID":"0b19bb5f-a003-4ea5-960c-0a1169f785db","Type":"ContainerDied","Data":"de6a54dfde49bdd8734ed02e2601eab87d8661f23da25fffbb69448c0cdcbf48"}
Sep 17 23:48:08.935817 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.935798    2562 scope.go:117] "RemoveContainer" containerID="b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da"
Sep 17 23:48:08.944835 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.944816    2562 scope.go:117] "RemoveContainer" containerID="e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8"
Sep 17 23:48:08.952721 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.952704    2562 scope.go:117] "RemoveContainer" containerID="c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c"
Sep 17 23:48:08.956128 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.956107    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"]
Sep 17 23:48:08.959951 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.959928    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dvprp9"]
Sep 17 23:48:08.961001 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.960984    2562 scope.go:117] "RemoveContainer" containerID="b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da"
Sep 17 23:48:08.961310 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:48:08.961289    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da\": container with ID starting with b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da not found: ID does not exist" containerID="b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da"
Sep 17 23:48:08.961380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.961319    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da"} err="failed to get container status \"b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da\": rpc error: code = NotFound desc = could not find container \"b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da\": container with ID starting with b628508ff5dbd76f05aff908ada7ab79931472ff619b53a23c01d2ee7084e9da not found: ID does not exist"
Sep 17 23:48:08.961380 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.961337    2562 scope.go:117] "RemoveContainer" containerID="e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8"
Sep 17 23:48:08.961626 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:48:08.961609    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8\": container with ID starting with e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8 not found: ID does not exist" containerID="e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8"
Sep 17 23:48:08.961692 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.961632    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8"} err="failed to get container status \"e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8\": rpc error: code = NotFound desc = could not find container \"e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8\": container with ID starting with e85ae6232202415887e1ceb7e54ef5f9ebbeb8f0ce40b9a04fb4fd77ab7f0cd8 not found: ID does not exist"
Sep 17 23:48:08.961692 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.961648    2562 scope.go:117] "RemoveContainer" containerID="c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c"
Sep 17 23:48:08.961874 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:48:08.961858    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c\": container with ID starting with c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c not found: ID does not exist" containerID="c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c"
Sep 17 23:48:08.961938 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:08.961878    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c"} err="failed to get container status \"c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c\": rpc error: code = NotFound desc = could not find container \"c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c\": container with ID starting with c778b5bf55415182c322b1222e4ba0bb0d1703b37b941cee162dfd14b513516c not found: ID does not exist"
Sep 17 23:48:09.420107 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:48:09.420074    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" path="/var/lib/kubelet/pods/0b19bb5f-a003-4ea5-960c-0a1169f785db/volumes"
Sep 17 23:49:00.590393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.590351    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"]
Sep 17 23:49:00.590870 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.590853    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="modelcar"
Sep 17 23:49:00.590940 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.590872    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="modelcar"
Sep 17 23:49:00.590940 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.590938    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="modelcar-init"
Sep 17 23:49:00.591008 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.590948    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="modelcar-init"
Sep 17 23:49:00.591008 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.590959    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main"
Sep 17 23:49:00.591008 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.590968    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main"
Sep 17 23:49:00.591109 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.591058    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="modelcar"
Sep 17 23:49:00.591109 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.591070    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b19bb5f-a003-4ea5-960c-0a1169f785db" containerName="main"
Sep 17 23:49:00.594651 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.594634    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.597269 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.597244    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Sep 17 23:49:00.597269 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.597229    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:49:00.597468 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.597244    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 17 23:49:00.597588 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.597571    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 17 23:49:00.598267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.598251    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-xlz6l\""
Sep 17 23:49:00.598379 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.598267    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 17 23:49:00.603589 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.603565    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"]
Sep 17 23:49:00.644724 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644678    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tls-certs\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.644724 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644721    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-model-cache\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.645000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644756    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.645000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644801    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tmp-dir\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.645000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644824    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-home\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.645000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644925    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-dshm\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.645000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644958    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.645000 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.644980    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lc55\" (UniqueName: \"kubernetes.io/projected/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kube-api-access-6lc55\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.745646 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-home\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.745828 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745672    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-dshm\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.745828 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745701    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.745828 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745730    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lc55\" (UniqueName: \"kubernetes.io/projected/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kube-api-access-6lc55\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.745828 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745769    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tls-certs\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.745828 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745803    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-model-cache\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.746151 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745848    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.746151 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.745893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tmp-dir\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.746151 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.746065    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-home\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.746311 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.746267    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tmp-dir\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.746387 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.746359    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-model-cache\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.746473 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.746403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.746684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.746668    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-cabundle-cert\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.748003 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.747982    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-dshm\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.748354 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.748335    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tls-certs\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.754076 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.754055    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lc55\" (UniqueName: \"kubernetes.io/projected/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kube-api-access-6lc55\") pod \"custom-route-timeout-test-kserve-7754cff758-fdgdj\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:00.906422 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:00.906335    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:01.182698 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:01.182666    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"]
Sep 17 23:49:01.184627 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:49:01.184599    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4ed415_9db7_49f3_bd50_e64f3f4957ad.slice/crio-4edc1e48806f4469e48797bb03f210b544eec1b0abbb514ae5db87c7b08e5a8f WatchSource:0}: Error finding container 4edc1e48806f4469e48797bb03f210b544eec1b0abbb514ae5db87c7b08e5a8f: Status 404 returned error can't find the container with id 4edc1e48806f4469e48797bb03f210b544eec1b0abbb514ae5db87c7b08e5a8f
Sep 17 23:49:01.206874 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:01.206847    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" event={"ID":"5a4ed415-9db7-49f3-bd50-e64f3f4957ad","Type":"ContainerStarted","Data":"4edc1e48806f4469e48797bb03f210b544eec1b0abbb514ae5db87c7b08e5a8f"}
Sep 17 23:49:02.212974 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:02.212934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" event={"ID":"5a4ed415-9db7-49f3-bd50-e64f3f4957ad","Type":"ContainerStarted","Data":"4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f"}
Sep 17 23:49:04.224312 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:04.224283    2562 generic.go:356] "Generic (PLEG): container finished" podID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerID="4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f" exitCode=0
Sep 17 23:49:04.224689 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:04.224361    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" event={"ID":"5a4ed415-9db7-49f3-bd50-e64f3f4957ad","Type":"ContainerDied","Data":"4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f"}
Sep 17 23:49:05.231542 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:05.231504    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" event={"ID":"5a4ed415-9db7-49f3-bd50-e64f3f4957ad","Type":"ContainerStarted","Data":"0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1"}
Sep 17 23:49:05.256332 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:05.256280    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podStartSLOduration=3.079165609 podStartE2EDuration="5.25626373s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.177098121s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:49:05.253955773 +0000 UTC m=+2306.348482807" watchObservedRunningTime="2026-09-17 23:49:05.25626373 +0000 UTC m=+2306.350790768"
Sep 17 23:49:05.907083 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:05.907049    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:10.906994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:10.906953    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:49:10.908630 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:10.908597    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:49:20.907758 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:20.907711    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:49:30.907204 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:30.907158    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:49:40.907598 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:40.907501    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:49:50.907150 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:49:50.907096    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:50:00.907678 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:50:00.907623    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:50:10.907757 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:50:10.907713    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:50:20.907616 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:50:20.907564    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:50:30.906874 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:50:30.906823    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:50:40.907220 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:50:40.907168    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:50:50.907093 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:50:50.907044    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:51:00.917462 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:00.917423    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:51:00.931671 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:00.931633    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:51:08.644009 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:08.643968    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"]
Sep 17 23:51:23.667664 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:23.667615    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" containerID="cri-o://0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1" gracePeriod=45
Sep 17 23:51:23.918544 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:23.918442    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Sep 17 23:51:24.410599 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.410573    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:51:24.484953 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.484830    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lc55\" (UniqueName: \"kubernetes.io/projected/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kube-api-access-6lc55\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.484953 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.484882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tls-certs\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.484953 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.484932    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tmp-dir\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.485255 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485033    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-home\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.485255 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485103    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-model-cache\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.485255 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485146    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-dshm\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.485255 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485232    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-cabundle-cert\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.485466 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kserve-provision-location\") pod \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\" (UID: \"5a4ed415-9db7-49f3-bd50-e64f3f4957ad\") "
Sep 17 23:51:24.485466 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485382    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:24.485640 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485605    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:24.485854 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485789    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-home" (OuterVolumeSpecName: "home") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:24.485854 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.485805    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:51:24.487447 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.487421    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:51:24.487562 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.487528    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kube-api-access-6lc55" (OuterVolumeSpecName: "kube-api-access-6lc55") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "kube-api-access-6lc55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:51:24.487741 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.487716    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-dshm" (OuterVolumeSpecName: "dshm") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:24.498544 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.498518    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:24.509016 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.508983    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5a4ed415-9db7-49f3-bd50-e64f3f4957ad" (UID: "5a4ed415-9db7-49f3-bd50-e64f3f4957ad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:51:24.586952 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.586925    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:24.586952 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.586949    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:24.586952 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.586959    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lc55\" (UniqueName: \"kubernetes.io/projected/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-kube-api-access-6lc55\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:24.587170 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.586969    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:24.587170 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.586977    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:24.587170 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.586984    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:24.587170 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:24.586991    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a4ed415-9db7-49f3-bd50-e64f3f4957ad-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:51:25.014134 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.014090    2562 generic.go:356] "Generic (PLEG): container finished" podID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerID="0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1" exitCode=0
Sep 17 23:51:25.014566 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.014140    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"
Sep 17 23:51:25.014566 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.014142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" event={"ID":"5a4ed415-9db7-49f3-bd50-e64f3f4957ad","Type":"ContainerDied","Data":"0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1"}
Sep 17 23:51:25.014566 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.014195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj" event={"ID":"5a4ed415-9db7-49f3-bd50-e64f3f4957ad","Type":"ContainerDied","Data":"4edc1e48806f4469e48797bb03f210b544eec1b0abbb514ae5db87c7b08e5a8f"}
Sep 17 23:51:25.014566 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.014217    2562 scope.go:117] "RemoveContainer" containerID="0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1"
Sep 17 23:51:25.023845 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.023828    2562 scope.go:117] "RemoveContainer" containerID="4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f"
Sep 17 23:51:25.031685 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.031668    2562 scope.go:117] "RemoveContainer" containerID="0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1"
Sep 17 23:51:25.031936 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:51:25.031919    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1\": container with ID starting with 0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1 not found: ID does not exist" containerID="0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1"
Sep 17 23:51:25.031994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.031945    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1"} err="failed to get container status \"0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1\": rpc error: code = NotFound desc = could not find container \"0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1\": container with ID starting with 0df5d28bf294d564dc8ed91220b0a46e1db4ae0c5aa8c42ef2d4db2211af55a1 not found: ID does not exist"
Sep 17 23:51:25.031994 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.031961    2562 scope.go:117] "RemoveContainer" containerID="4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f"
Sep 17 23:51:25.032207 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:51:25.032192    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f\": container with ID starting with 4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f not found: ID does not exist" containerID="4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f"
Sep 17 23:51:25.032259 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.032210    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f"} err="failed to get container status \"4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f\": rpc error: code = NotFound desc = could not find container \"4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f\": container with ID starting with 4dc78368b27535d6e81d3f077489dfa093c82564b5365294f190cb2a8b061d7f not found: ID does not exist"
Sep 17 23:51:25.036713 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.036692    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"]
Sep 17 23:51:25.039969 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.039936    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-7754cff758-fdgdj"]
Sep 17 23:51:25.419941 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:51:25.419890    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" path="/var/lib/kubelet/pods/5a4ed415-9db7-49f3-bd50-e64f3f4957ad/volumes"
Sep 17 23:52:10.089668 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.089633    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"]
Sep 17 23:52:10.090223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.089976    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main"
Sep 17 23:52:10.090223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.089987    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main"
Sep 17 23:52:10.090223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.090006    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="storage-initializer"
Sep 17 23:52:10.090223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.090012    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="storage-initializer"
Sep 17 23:52:10.090223 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.090069    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a4ed415-9db7-49f3-bd50-e64f3f4957ad" containerName="main"
Sep 17 23:52:10.093202 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.093182    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.096364 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.096323    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Sep 17 23:52:10.096503 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.096374    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 17 23:52:10.096503 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.096390    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-48bfg\""
Sep 17 23:52:10.096503 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.096414    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 17 23:52:10.096503 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.096456    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 17 23:52:10.096503 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.096341    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:52:10.104827 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.104799    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"]
Sep 17 23:52:10.183625 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183587    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-tmp-dir\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.183805 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183644    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f11145a-5481-496c-8138-73aeeb3de7f1-tls-certs\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.183805 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183677    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-kserve-provision-location\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.183805 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183716    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-home\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.183805 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183748    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-model-cache\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.183805 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183774    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f11145a-5481-496c-8138-73aeeb3de7f1-cabundle-cert\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.184009 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183816    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-dshm\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.184009 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.183859    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6rn\" (UniqueName: \"kubernetes.io/projected/7f11145a-5481-496c-8138-73aeeb3de7f1-kube-api-access-5w6rn\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.284947 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.284884    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-dshm\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285135 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.284952    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6rn\" (UniqueName: \"kubernetes.io/projected/7f11145a-5481-496c-8138-73aeeb3de7f1-kube-api-access-5w6rn\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285135 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-tmp-dir\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285135 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285068    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f11145a-5481-496c-8138-73aeeb3de7f1-tls-certs\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285135 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-kserve-provision-location\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285352 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285218    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-home\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285352 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-model-cache\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285352 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285273    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f11145a-5481-496c-8138-73aeeb3de7f1-cabundle-cert\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285477 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285431    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-kserve-provision-location\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285585 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285564    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-home\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285588    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-tmp-dir\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285598    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-model-cache\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.285893 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.285874    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f11145a-5481-496c-8138-73aeeb3de7f1-cabundle-cert\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.287233 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.287210    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-dshm\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.287602 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.287582    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f11145a-5481-496c-8138-73aeeb3de7f1-tls-certs\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.292523 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.292504    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6rn\" (UniqueName: \"kubernetes.io/projected/7f11145a-5481-496c-8138-73aeeb3de7f1-kube-api-access-5w6rn\") pod \"router-with-refs-test-kserve-5d8c657c8b-zn7mz\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.404743 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.404657    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:10.686021 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:10.685993    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"]
Sep 17 23:52:10.688091 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:52:10.688058    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f11145a_5481_496c_8138_73aeeb3de7f1.slice/crio-63cda95af7e42b4a812c538840d4c7475fb1dd085b4285c66b4755e5d7fda7eb WatchSource:0}: Error finding container 63cda95af7e42b4a812c538840d4c7475fb1dd085b4285c66b4755e5d7fda7eb: Status 404 returned error can't find the container with id 63cda95af7e42b4a812c538840d4c7475fb1dd085b4285c66b4755e5d7fda7eb
Sep 17 23:52:11.248264 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:11.248211    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" event={"ID":"7f11145a-5481-496c-8138-73aeeb3de7f1","Type":"ContainerStarted","Data":"298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee"}
Sep 17 23:52:11.248705 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:11.248289    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" event={"ID":"7f11145a-5481-496c-8138-73aeeb3de7f1","Type":"ContainerStarted","Data":"63cda95af7e42b4a812c538840d4c7475fb1dd085b4285c66b4755e5d7fda7eb"}
Sep 17 23:52:13.259540 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:13.259506    2562 generic.go:356] "Generic (PLEG): container finished" podID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerID="298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee" exitCode=0
Sep 17 23:52:13.260008 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:13.259557    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" event={"ID":"7f11145a-5481-496c-8138-73aeeb3de7f1","Type":"ContainerDied","Data":"298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee"}
Sep 17 23:52:14.266489 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:14.266453    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" event={"ID":"7f11145a-5481-496c-8138-73aeeb3de7f1","Type":"ContainerStarted","Data":"afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f"}
Sep 17 23:52:14.290282 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:14.290209    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podStartSLOduration=2.005322567 podStartE2EDuration="4.290183299s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.284860732s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:52:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:52:14.28829836 +0000 UTC m=+2495.382825397" watchObservedRunningTime="2026-09-17 23:52:14.290183299 +0000 UTC m=+2495.384710339"
Sep 17 23:52:14.405754 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:14.405714    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:20.404829 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:20.404790    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:52:20.406478 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:20.406441    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:52:30.406082 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:30.406027    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:52:40.406049 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:40.405944    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:52:50.406127 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:52:50.406086    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:53:00.405925 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:53:00.405859    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:53:10.405611 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:53:10.405559    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:53:20.406079 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:53:20.406036    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:53:30.405071 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:53:30.405027    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:53:40.405192 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:53:40.405151    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:53:50.405810 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:53:50.405764    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:54:00.405234 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:54:00.405181    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:54:10.415540 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:54:10.415503    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:54:10.423684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:54:10.423659    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:55:36.913378 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.913290    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"]
Sep 17 23:55:36.917647 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.917630    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.920573 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.920551    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-l7n9m\""
Sep 17 23:55:36.920709 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.920551    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 17 23:55:36.920709 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.920656    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 17 23:55:36.920894 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.920877    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Sep 17 23:55:36.920999 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.920952    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:55:36.921054 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.921003    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 17 23:55:36.926554 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.926528    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"]
Sep 17 23:55:36.972111 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-model-cache\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.972111 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972111    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b92466-ef90-4f5b-a8a3-a580e092936e-tls-certs\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.972314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-kserve-provision-location\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.972314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972195    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-home\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.972314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-dshm\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.972314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972237    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-tmp-dir\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.972314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972253    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b92466-ef90-4f5b-a8a3-a580e092936e-cabundle-cert\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:36.972314 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:36.972291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsn4\" (UniqueName: \"kubernetes.io/projected/51b92466-ef90-4f5b-a8a3-a580e092936e-kube-api-access-ntsn4\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.072761 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.072723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-model-cache\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.072761 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.072767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b92466-ef90-4f5b-a8a3-a580e092936e-tls-certs\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.072805    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-kserve-provision-location\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.072853    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-home\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.072877    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-dshm\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.072961    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-tmp-dir\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.073007    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b92466-ef90-4f5b-a8a3-a580e092936e-cabundle-cert\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073291 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.073052    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsn4\" (UniqueName: \"kubernetes.io/projected/51b92466-ef90-4f5b-a8a3-a580e092936e-kube-api-access-ntsn4\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073291 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.073227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-model-cache\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073291 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.073275    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-kserve-provision-location\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073452 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.073421    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-tmp-dir\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073635 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.073616    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-home\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.073750 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.073731    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b92466-ef90-4f5b-a8a3-a580e092936e-cabundle-cert\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.075155 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.075137    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-dshm\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.075430 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.075411    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b92466-ef90-4f5b-a8a3-a580e092936e-tls-certs\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.081253 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.081227    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsn4\" (UniqueName: \"kubernetes.io/projected/51b92466-ef90-4f5b-a8a3-a580e092936e-kube-api-access-ntsn4\") pod \"stop-feature-test-kserve-54c98ddcdd-pbpzp\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.228767 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.228673    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:37.511891 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:37.511856    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"]
Sep 17 23:55:37.513227 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:55:37.513197    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b92466_ef90_4f5b_a8a3_a580e092936e.slice/crio-253b30d6b29be88b3a82399274acd961e8498285f1d2d6757e724603b875d2e9 WatchSource:0}: Error finding container 253b30d6b29be88b3a82399274acd961e8498285f1d2d6757e724603b875d2e9: Status 404 returned error can't find the container with id 253b30d6b29be88b3a82399274acd961e8498285f1d2d6757e724603b875d2e9
Sep 17 23:55:38.375826 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:38.375789    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" event={"ID":"51b92466-ef90-4f5b-a8a3-a580e092936e","Type":"ContainerStarted","Data":"c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd"}
Sep 17 23:55:38.376245 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:38.375841    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" event={"ID":"51b92466-ef90-4f5b-a8a3-a580e092936e","Type":"ContainerStarted","Data":"253b30d6b29be88b3a82399274acd961e8498285f1d2d6757e724603b875d2e9"}
Sep 17 23:55:40.388498 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:40.388458    2562 generic.go:356] "Generic (PLEG): container finished" podID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerID="c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd" exitCode=0
Sep 17 23:55:40.388950 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:40.388533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" event={"ID":"51b92466-ef90-4f5b-a8a3-a580e092936e","Type":"ContainerDied","Data":"c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd"}
Sep 17 23:55:41.396398 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:41.396359    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" event={"ID":"51b92466-ef90-4f5b-a8a3-a580e092936e","Type":"ContainerStarted","Data":"50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef"}
Sep 17 23:55:41.420682 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:41.420622    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podStartSLOduration=3.13544045 podStartE2EDuration="5.420604701s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.285164251s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:55:41.417360717 +0000 UTC m=+2702.511887753" watchObservedRunningTime="2026-09-17 23:55:41.420604701 +0000 UTC m=+2702.515131736"
Sep 17 23:55:42.229333 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:42.229281    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:47.229469 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:47.229427    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:55:47.231150 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:47.231123    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:55:57.229554 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:55:57.229501    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:56:07.229402 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:56:07.229353    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:56:17.230102 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:56:17.230058    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:56:27.229520 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:56:27.229465    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:56:37.229534 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:56:37.229483    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:56:47.229564 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:56:47.229511    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:56:57.229955 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:56:57.229869    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:57:07.230160 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:07.230108    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:57:17.229822 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:17.229767    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:57:27.229092 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:27.229046    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:57:37.239948 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:37.239884    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:57:37.247863 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:37.247831    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:57:38.854669 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:38.854629    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"]
Sep 17 23:57:54.086789 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.086715    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" containerID="cri-o://50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef" gracePeriod=45
Sep 17 23:57:54.240736 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.240685    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 17 23:57:54.836506 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.836480    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:57:54.912455 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912358    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b92466-ef90-4f5b-a8a3-a580e092936e-cabundle-cert\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.912455 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912422    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-home\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.912455 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912449    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-tmp-dir\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.912731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912479    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntsn4\" (UniqueName: \"kubernetes.io/projected/51b92466-ef90-4f5b-a8a3-a580e092936e-kube-api-access-ntsn4\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.912731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912513    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b92466-ef90-4f5b-a8a3-a580e092936e-tls-certs\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.912731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912586    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-model-cache\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.912731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912634    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-kserve-provision-location\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.912731 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912658    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-dshm\") pod \"51b92466-ef90-4f5b-a8a3-a580e092936e\" (UID: \"51b92466-ef90-4f5b-a8a3-a580e092936e\") "
Sep 17 23:57:54.913024 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912815    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b92466-ef90-4f5b-a8a3-a580e092936e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:57:54.913024 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.912986    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/51b92466-ef90-4f5b-a8a3-a580e092936e-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:54.913173 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.913142    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-model-cache" (OuterVolumeSpecName: "model-cache") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:57:54.913379 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.913225    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-home" (OuterVolumeSpecName: "home") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:57:54.915376 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.915353    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-dshm" (OuterVolumeSpecName: "dshm") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:57:54.915464 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.915411    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b92466-ef90-4f5b-a8a3-a580e092936e-kube-api-access-ntsn4" (OuterVolumeSpecName: "kube-api-access-ntsn4") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "kube-api-access-ntsn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:57:54.915784 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.915760    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b92466-ef90-4f5b-a8a3-a580e092936e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:57:54.925465 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.925437    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:57:54.934375 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:54.934311    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51b92466-ef90-4f5b-a8a3-a580e092936e" (UID: "51b92466-ef90-4f5b-a8a3-a580e092936e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:57:55.013549 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.013507    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:55.013549 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.013548    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:55.013759 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.013562    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:55.013759 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.013577    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntsn4\" (UniqueName: \"kubernetes.io/projected/51b92466-ef90-4f5b-a8a3-a580e092936e-kube-api-access-ntsn4\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:55.013759 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.013591    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b92466-ef90-4f5b-a8a3-a580e092936e-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:55.013759 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.013605    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:55.013759 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.013618    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51b92466-ef90-4f5b-a8a3-a580e092936e-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:57:55.148064 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.148027    2562 generic.go:356] "Generic (PLEG): container finished" podID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerID="50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef" exitCode=0
Sep 17 23:57:55.148460 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.148080    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"
Sep 17 23:57:55.148460 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.148116    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" event={"ID":"51b92466-ef90-4f5b-a8a3-a580e092936e","Type":"ContainerDied","Data":"50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef"}
Sep 17 23:57:55.148460 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.148179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp" event={"ID":"51b92466-ef90-4f5b-a8a3-a580e092936e","Type":"ContainerDied","Data":"253b30d6b29be88b3a82399274acd961e8498285f1d2d6757e724603b875d2e9"}
Sep 17 23:57:55.148460 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.148210    2562 scope.go:117] "RemoveContainer" containerID="50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef"
Sep 17 23:57:55.158568 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.158548    2562 scope.go:117] "RemoveContainer" containerID="c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd"
Sep 17 23:57:55.167365 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.167344    2562 scope.go:117] "RemoveContainer" containerID="50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef"
Sep 17 23:57:55.167686 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:57:55.167665    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef\": container with ID starting with 50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef not found: ID does not exist" containerID="50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef"
Sep 17 23:57:55.167763 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.167696    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef"} err="failed to get container status \"50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef\": rpc error: code = NotFound desc = could not find container \"50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef\": container with ID starting with 50340dc2a085fab37069c46c60aa54a675fb64238310fd49747a16327ffa42ef not found: ID does not exist"
Sep 17 23:57:55.167763 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.167715    2562 scope.go:117] "RemoveContainer" containerID="c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd"
Sep 17 23:57:55.168017 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:57:55.168001    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd\": container with ID starting with c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd not found: ID does not exist" containerID="c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd"
Sep 17 23:57:55.168069 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.168021    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd"} err="failed to get container status \"c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd\": rpc error: code = NotFound desc = could not find container \"c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd\": container with ID starting with c88f9351b76fcb730c4607b75d2b9b2a4a4c0d9000677ff6f0a0f54379c6aafd not found: ID does not exist"
Sep 17 23:57:55.171645 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.171621    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"]
Sep 17 23:57:55.177393 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.177367    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-54c98ddcdd-pbpzp"]
Sep 17 23:57:55.419518 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:57:55.419436    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" path="/var/lib/kubelet/pods/51b92466-ef90-4f5b-a8a3-a580e092936e/volumes"
Sep 17 23:58:19.841545 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:19.841510    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"]
Sep 17 23:58:34.866867 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:34.866759    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" containerID="cri-o://afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f" gracePeriod=45
Sep 17 23:58:35.416973 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.416943    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Sep 17 23:58:35.623835 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.623809    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:58:35.767587 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767487    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f11145a-5481-496c-8138-73aeeb3de7f1-cabundle-cert\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.767587 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767543    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w6rn\" (UniqueName: \"kubernetes.io/projected/7f11145a-5481-496c-8138-73aeeb3de7f1-kube-api-access-5w6rn\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.767587 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767578    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-dshm\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.767879 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f11145a-5481-496c-8138-73aeeb3de7f1-tls-certs\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.767879 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767640    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-kserve-provision-location\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.767879 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767722    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-home\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.767879 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767749    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-tmp-dir\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.767879 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.767770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-model-cache\") pod \"7f11145a-5481-496c-8138-73aeeb3de7f1\" (UID: \"7f11145a-5481-496c-8138-73aeeb3de7f1\") "
Sep 17 23:58:35.768177 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.768005    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f11145a-5481-496c-8138-73aeeb3de7f1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 23:58:35.768177 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.768096    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7f11145a-5481-496c-8138-73aeeb3de7f1-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.768299 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.768235    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:35.768517 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.768493    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-home" (OuterVolumeSpecName: "home") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:35.770537 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.770485    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f11145a-5481-496c-8138-73aeeb3de7f1-kube-api-access-5w6rn" (OuterVolumeSpecName: "kube-api-access-5w6rn") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "kube-api-access-5w6rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 23:58:35.770537 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.770489    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f11145a-5481-496c-8138-73aeeb3de7f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 23:58:35.770735 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.770710    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-dshm" (OuterVolumeSpecName: "dshm") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:35.782384 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.782347    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:35.791831 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.791798    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f11145a-5481-496c-8138-73aeeb3de7f1" (UID: "7f11145a-5481-496c-8138-73aeeb3de7f1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 23:58:35.869578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.869537    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w6rn\" (UniqueName: \"kubernetes.io/projected/7f11145a-5481-496c-8138-73aeeb3de7f1-kube-api-access-5w6rn\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.869578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.869570    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.869578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.869581    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f11145a-5481-496c-8138-73aeeb3de7f1-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.869578 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.869590    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.870106 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.869598    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.870106 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.869606    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:35.870106 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:35.869613    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f11145a-5481-496c-8138-73aeeb3de7f1-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 17 23:58:36.363130 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.363096    2562 generic.go:356] "Generic (PLEG): container finished" podID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerID="afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f" exitCode=0
Sep 17 23:58:36.363302 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.363163    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"
Sep 17 23:58:36.363302 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.363188    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" event={"ID":"7f11145a-5481-496c-8138-73aeeb3de7f1","Type":"ContainerDied","Data":"afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f"}
Sep 17 23:58:36.363302 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.363242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz" event={"ID":"7f11145a-5481-496c-8138-73aeeb3de7f1","Type":"ContainerDied","Data":"63cda95af7e42b4a812c538840d4c7475fb1dd085b4285c66b4755e5d7fda7eb"}
Sep 17 23:58:36.363302 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.363264    2562 scope.go:117] "RemoveContainer" containerID="afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f"
Sep 17 23:58:36.372509 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.372492    2562 scope.go:117] "RemoveContainer" containerID="298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee"
Sep 17 23:58:36.380696 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.380677    2562 scope.go:117] "RemoveContainer" containerID="afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f"
Sep 17 23:58:36.380973 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:58:36.380955    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f\": container with ID starting with afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f not found: ID does not exist" containerID="afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f"
Sep 17 23:58:36.381023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.380981    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f"} err="failed to get container status \"afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f\": rpc error: code = NotFound desc = could not find container \"afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f\": container with ID starting with afcb6903f1cddeebdb41b497ad627336d07d6c58af1c75f7dfd9739d746de87f not found: ID does not exist"
Sep 17 23:58:36.381023 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.380998    2562 scope.go:117] "RemoveContainer" containerID="298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee"
Sep 17 23:58:36.381268 ip-10-0-132-155 kubenswrapper[2562]: E0917 23:58:36.381249    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee\": container with ID starting with 298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee not found: ID does not exist" containerID="298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee"
Sep 17 23:58:36.381310 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.381275    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee"} err="failed to get container status \"298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee\": rpc error: code = NotFound desc = could not find container \"298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee\": container with ID starting with 298c077aaf0b320d3f188f3dd52e20dd38b8990a3b68d72d04635c503d1d9bee not found: ID does not exist"
Sep 17 23:58:36.386407 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.386384    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"]
Sep 17 23:58:36.391168 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:36.391147    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5d8c657c8b-zn7mz"]
Sep 17 23:58:37.418520 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:58:37.418480    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" path="/var/lib/kubelet/pods/7f11145a-5481-496c-8138-73aeeb3de7f1/volumes"
Sep 17 23:59:38.283830 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.283790    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"]
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284187    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main"
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284203    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main"
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284212    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="storage-initializer"
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284218    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="storage-initializer"
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284224    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main"
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284230    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main"
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284238    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="storage-initializer"
Sep 17 23:59:38.284267 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284243    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="storage-initializer"
Sep 17 23:59:38.284517 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284315    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f11145a-5481-496c-8138-73aeeb3de7f1" containerName="main"
Sep 17 23:59:38.284517 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.284324    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b92466-ef90-4f5b-a8a3-a580e092936e" containerName="main"
Sep 17 23:59:38.287513 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.287489    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.290511 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.290465    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 17 23:59:38.290511 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.290465    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-nv8nh\""
Sep 17 23:59:38.290511 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.290511    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 23:59:38.290714 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.290517    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-4wffp\""
Sep 17 23:59:38.290714 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.290639    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 17 23:59:38.290795 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.290761    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 17 23:59:38.291416 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.291401    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Sep 17 23:59:38.298773 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.298752    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"]
Sep 17 23:59:38.388309 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388283    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.388449 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388315    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.388449 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388340    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.388449 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388367    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wdrg\" (UniqueName: \"kubernetes.io/projected/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kube-api-access-6wdrg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.388560 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388467    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.388560 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.388560 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388529    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.388660 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.388567    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.483824 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.483797    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"]
Sep 17 23:59:38.487401 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.487387    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.489632 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489607    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.489730 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wdrg\" (UniqueName: \"kubernetes.io/projected/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kube-api-access-6wdrg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.489730 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.489730 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.489869 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489759    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.489869 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489796    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.489869 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489821    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.489869 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.489850    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.490308 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.490287    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.490424 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.490403    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.490535 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.490513    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.490641 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.490535    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.490701 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.490654    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.492129 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.492112    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.492603 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.492587    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.500053 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.500031    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"]
Sep 17 23:59:38.502694 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.502664    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wdrg\" (UniqueName: \"kubernetes.io/projected/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kube-api-access-6wdrg\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.590830 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.590803    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056087a0-c8df-4f51-8561-7dc0324bcade-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.591022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.590848    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056087a0-c8df-4f51-8561-7dc0324bcade-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.591022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.590871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.591022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.590889    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4mxw\" (UniqueName: \"kubernetes.io/projected/056087a0-c8df-4f51-8561-7dc0324bcade-kube-api-access-w4mxw\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.591022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.590952    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.591022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.590968    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.591022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.590990    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.591256 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.591085    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.597210 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.597193    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:38.691611 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.691584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056087a0-c8df-4f51-8561-7dc0324bcade-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.691788 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.691618    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.691788 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.691640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4mxw\" (UniqueName: \"kubernetes.io/projected/056087a0-c8df-4f51-8561-7dc0324bcade-kube-api-access-w4mxw\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.691788 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.691669    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.691990 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.691927    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.691990 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.691984    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.692098 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.692021    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.692098 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.692031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.692098 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.692080    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.692276 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.692086    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056087a0-c8df-4f51-8561-7dc0324bcade-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.692276 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.692233    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.692386 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.692354    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.692783 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.692759    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056087a0-c8df-4f51-8561-7dc0324bcade-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.694214 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.694192    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.694372 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.694358    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056087a0-c8df-4f51-8561-7dc0324bcade-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.699401 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.699376    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4mxw\" (UniqueName: \"kubernetes.io/projected/056087a0-c8df-4f51-8561-7dc0324bcade-kube-api-access-w4mxw\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.814997 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.814956    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:38.878247 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:38.878218    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"]
Sep 17 23:59:38.881289 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:59:38.881258    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8fe9c7_8688_4705_96b9_eb19d1b1e948.slice/crio-b18c17fd30cfdfa86bf72d3168f593ec9b1badaea71a914d493e35911f62fe9a WatchSource:0}: Error finding container b18c17fd30cfdfa86bf72d3168f593ec9b1badaea71a914d493e35911f62fe9a: Status 404 returned error can't find the container with id b18c17fd30cfdfa86bf72d3168f593ec9b1badaea71a914d493e35911f62fe9a
Sep 17 23:59:39.091684 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:39.091656    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"]
Sep 17 23:59:39.093613 ip-10-0-132-155 kubenswrapper[2562]: W0917 23:59:39.093586    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056087a0_c8df_4f51_8561_7dc0324bcade.slice/crio-63b648370329f55faeed3bc9f904c5e81d30033b0e966ed8a636084efcbcf7a0 WatchSource:0}: Error finding container 63b648370329f55faeed3bc9f904c5e81d30033b0e966ed8a636084efcbcf7a0: Status 404 returned error can't find the container with id 63b648370329f55faeed3bc9f904c5e81d30033b0e966ed8a636084efcbcf7a0
Sep 17 23:59:39.691824 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:39.691768    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerStarted","Data":"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"}
Sep 17 23:59:39.692290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:39.691846    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerStarted","Data":"b18c17fd30cfdfa86bf72d3168f593ec9b1badaea71a914d493e35911f62fe9a"}
Sep 17 23:59:39.692290 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:39.691876    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:39.693560 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:39.693528    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" event={"ID":"056087a0-c8df-4f51-8561-7dc0324bcade","Type":"ContainerStarted","Data":"d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6"}
Sep 17 23:59:39.693653 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:39.693588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" event={"ID":"056087a0-c8df-4f51-8561-7dc0324bcade","Type":"ContainerStarted","Data":"63b648370329f55faeed3bc9f904c5e81d30033b0e966ed8a636084efcbcf7a0"}
Sep 17 23:59:40.702520 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:40.702470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerStarted","Data":"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"}
Sep 17 23:59:41.709928 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:41.709879    2562 generic.go:356] "Generic (PLEG): container finished" podID="056087a0-c8df-4f51-8561-7dc0324bcade" containerID="d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6" exitCode=0
Sep 17 23:59:41.710296 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:41.709933    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" event={"ID":"056087a0-c8df-4f51-8561-7dc0324bcade","Type":"ContainerDied","Data":"d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6"}
Sep 17 23:59:42.717167 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:42.717130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" event={"ID":"056087a0-c8df-4f51-8561-7dc0324bcade","Type":"ContainerStarted","Data":"0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36"}
Sep 17 23:59:42.719026 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:42.718999    2562 generic.go:356] "Generic (PLEG): container finished" podID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerID="e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68" exitCode=0
Sep 17 23:59:42.719139 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:42.719046    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerDied","Data":"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"}
Sep 17 23:59:42.739260 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:42.739210    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podStartSLOduration=2.331785459 podStartE2EDuration="4.739188239s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.40740278s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:59:42.737195546 +0000 UTC m=+2943.831722582" watchObservedRunningTime="2026-09-17 23:59:42.739188239 +0000 UTC m=+2943.833715276"
Sep 17 23:59:43.728630 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:43.728586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerStarted","Data":"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"}
Sep 17 23:59:43.758197 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:43.758129    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podStartSLOduration=3.343334498 podStartE2EDuration="5.758105747s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.414771249s" isStatefulPod=false podCreationTimestamp="2026-09-17 23:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 23:59:43.755375065 +0000 UTC m=+2944.849902158" watchObservedRunningTime="2026-09-17 23:59:43.758105747 +0000 UTC m=+2944.852632787"
Sep 17 23:59:48.597649 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:48.597604    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:48.598251 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:48.597687    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:48.599308 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:48.599275    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 23:59:48.616519 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:48.616492    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 17 23:59:48.815710 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:48.815671    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:48.816022 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:48.815999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 17 23:59:48.817675 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:48.817648    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 17 23:59:58.598599 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:58.598549    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 17 23:59:58.815458 ip-10-0-132-155 kubenswrapper[2562]: I0917 23:59:58.815411    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:00:08.598747 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:08.598673    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:00:08.816013 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:08.815969    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:00:18.598738 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:18.598693    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:00:18.816436 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:18.816386    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:00:28.597962 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:28.597885    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:00:28.815966 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:28.815913    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:00:38.597768 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:38.597723    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:00:38.815802 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:38.815761    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:00:48.598350 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:48.598299    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:00:48.815538 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:48.815497    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:00:58.598628 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:58.598582    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:00:58.815632 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:00:58.815586    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:01:08.598556 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:08.598504    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:01:08.816212 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:08.816157    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:01:18.598673 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:18.598622    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:01:18.816319 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:18.816272    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:01:28.598153 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:28.598100    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:01:28.816258 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:28.816195    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:01:38.598318 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:38.598200    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:01:38.816186 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:38.816134    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:01:48.598703 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:48.598645    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:01:48.816184 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:48.816128    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:01:58.597998 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:58.597939    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:01:58.816539 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:01:58.816489    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:02:08.598156 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:08.598098    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:02:08.815650 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:08.815594    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:02:18.598343 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:18.598288    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Sep 18 00:02:18.816438 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:18.816378    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Sep 18 00:02:28.607319 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:28.607282    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 18 00:02:28.619017 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:28.618987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 18 00:02:28.825775 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:28.825745    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 18 00:02:28.833500 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:28.833473    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 18 00:02:38.371024 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:38.370987    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"]
Sep 18 00:02:38.377805 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:38.377773    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"]
Sep 18 00:02:53.400876 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:53.400825    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main" containerID="cri-o://0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36" gracePeriod=45
Sep 18 00:02:53.401450 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:53.401003    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main" containerID="cri-o://66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68" gracePeriod=45
Sep 18 00:02:54.216482 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.216446    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="llm-d-routing-sidecar" containerID="cri-o://460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44" gracePeriod=45
Sep 18 00:02:54.355348 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.355321    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 18 00:02:54.444160 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444125    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056087a0-c8df-4f51-8561-7dc0324bcade-tls-certs\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444160 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444160    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4mxw\" (UniqueName: \"kubernetes.io/projected/056087a0-c8df-4f51-8561-7dc0324bcade-kube-api-access-w4mxw\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444186    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-model-cache\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444202    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-dshm\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444261    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-kserve-provision-location\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444299    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-home\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444355    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-tmp-dir\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444397    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056087a0-c8df-4f51-8561-7dc0324bcade-cabundle-cert\") pod \"056087a0-c8df-4f51-8561-7dc0324bcade\" (UID: \"056087a0-c8df-4f51-8561-7dc0324bcade\") "
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444491    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-model-cache" (OuterVolumeSpecName: "model-cache") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.444697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444697    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.445177 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.444818    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056087a0-c8df-4f51-8561-7dc0324bcade-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:02:54.445177 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.445118    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-home" (OuterVolumeSpecName: "home") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.447449 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.447419    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-dshm" (OuterVolumeSpecName: "dshm") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.447565 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.447447    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056087a0-c8df-4f51-8561-7dc0324bcade-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:02:54.447565 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.447480    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056087a0-c8df-4f51-8561-7dc0324bcade-kube-api-access-w4mxw" (OuterVolumeSpecName: "kube-api-access-w4mxw") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "kube-api-access-w4mxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:02:54.464213 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.464159    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.467984 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.467946    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "056087a0-c8df-4f51-8561-7dc0324bcade" (UID: "056087a0-c8df-4f51-8561-7dc0324bcade"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.479343 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.479318    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 18 00:02:54.546148 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.546108    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.546148 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.546140    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/056087a0-c8df-4f51-8561-7dc0324bcade-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.546148 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.546148    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/056087a0-c8df-4f51-8561-7dc0324bcade-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.546148 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.546157    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4mxw\" (UniqueName: \"kubernetes.io/projected/056087a0-c8df-4f51-8561-7dc0324bcade-kube-api-access-w4mxw\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.546446 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.546165    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.546446 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.546173    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.546446 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.546180    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/056087a0-c8df-4f51-8561-7dc0324bcade-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.646568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646458    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-model-cache\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646526    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-home\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646576    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-cabundle-cert\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646858 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tmp-dir\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646858 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-dshm\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646858 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646679    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wdrg\" (UniqueName: \"kubernetes.io/projected/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kube-api-access-6wdrg\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646858 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646703    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tls-certs\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646858 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646771    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kserve-provision-location\") pod \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\" (UID: \"9f8fe9c7-8688-4705-96b9-eb19d1b1e948\") "
Sep 18 00:02:54.646858 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.646769    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.647201 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.647056    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.647201 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.647088    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:02:54.647367 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.647342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-home" (OuterVolumeSpecName: "home") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.648932 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.648891    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-dshm" (OuterVolumeSpecName: "dshm") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.648932 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.648896    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:02:54.649319 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.649294    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kube-api-access-6wdrg" (OuterVolumeSpecName: "kube-api-access-6wdrg") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "kube-api-access-6wdrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:02:54.658420 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.658400    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.670972 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.670940    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f8fe9c7-8688-4705-96b9-eb19d1b1e948" (UID: "9f8fe9c7-8688-4705-96b9-eb19d1b1e948"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:02:54.747676 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.747646    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.747676 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.747676    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.747859 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.747689    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.747859 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.747699    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.747859 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.747706    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.747859 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.747715    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wdrg\" (UniqueName: \"kubernetes.io/projected/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-kube-api-access-6wdrg\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.747859 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.747722    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f8fe9c7-8688-4705-96b9-eb19d1b1e948-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:02:54.838198 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.838159    2562 generic.go:356] "Generic (PLEG): container finished" podID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerID="66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68" exitCode=0
Sep 18 00:02:54.838198 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.838189    2562 generic.go:356] "Generic (PLEG): container finished" podID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerID="460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44" exitCode=0
Sep 18 00:02:54.838454 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.838238    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"
Sep 18 00:02:54.838454 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.838241    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerDied","Data":"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"}
Sep 18 00:02:54.838454 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.838325    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerDied","Data":"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"}
Sep 18 00:02:54.838454 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.838349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb" event={"ID":"9f8fe9c7-8688-4705-96b9-eb19d1b1e948","Type":"ContainerDied","Data":"b18c17fd30cfdfa86bf72d3168f593ec9b1badaea71a914d493e35911f62fe9a"}
Sep 18 00:02:54.838454 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.838379    2562 scope.go:117] "RemoveContainer" containerID="66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"
Sep 18 00:02:54.840045 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.840017    2562 generic.go:356] "Generic (PLEG): container finished" podID="056087a0-c8df-4f51-8561-7dc0324bcade" containerID="0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36" exitCode=0
Sep 18 00:02:54.840164 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.840086    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"
Sep 18 00:02:54.840164 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.840098    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" event={"ID":"056087a0-c8df-4f51-8561-7dc0324bcade","Type":"ContainerDied","Data":"0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36"}
Sep 18 00:02:54.840164 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.840142    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4" event={"ID":"056087a0-c8df-4f51-8561-7dc0324bcade","Type":"ContainerDied","Data":"63b648370329f55faeed3bc9f904c5e81d30033b0e966ed8a636084efcbcf7a0"}
Sep 18 00:02:54.849392 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.849377    2562 scope.go:117] "RemoveContainer" containerID="e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"
Sep 18 00:02:54.858769 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.858745    2562 scope.go:117] "RemoveContainer" containerID="460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"
Sep 18 00:02:54.862196 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.862167    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"]
Sep 18 00:02:54.868859 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.868838    2562 scope.go:117] "RemoveContainer" containerID="66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"
Sep 18 00:02:54.868991 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.868863    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-788d659d4bf4vnb"]
Sep 18 00:02:54.869211 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:02:54.869188    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68\": container with ID starting with 66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68 not found: ID does not exist" containerID="66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"
Sep 18 00:02:54.869277 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.869224    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"} err="failed to get container status \"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68\": rpc error: code = NotFound desc = could not find container \"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68\": container with ID starting with 66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68 not found: ID does not exist"
Sep 18 00:02:54.869277 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.869250    2562 scope.go:117] "RemoveContainer" containerID="e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"
Sep 18 00:02:54.869554 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:02:54.869538    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68\": container with ID starting with e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68 not found: ID does not exist" containerID="e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"
Sep 18 00:02:54.869607 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.869560    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"} err="failed to get container status \"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68\": rpc error: code = NotFound desc = could not find container \"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68\": container with ID starting with e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68 not found: ID does not exist"
Sep 18 00:02:54.869607 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.869575    2562 scope.go:117] "RemoveContainer" containerID="460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"
Sep 18 00:02:54.869808 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:02:54.869787    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44\": container with ID starting with 460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44 not found: ID does not exist" containerID="460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"
Sep 18 00:02:54.869862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.869813    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"} err="failed to get container status \"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44\": rpc error: code = NotFound desc = could not find container \"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44\": container with ID starting with 460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44 not found: ID does not exist"
Sep 18 00:02:54.869862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.869828    2562 scope.go:117] "RemoveContainer" containerID="66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"
Sep 18 00:02:54.870080 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.870061    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68"} err="failed to get container status \"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68\": rpc error: code = NotFound desc = could not find container \"66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68\": container with ID starting with 66411a0c72af5fd3785beb9af5f8166944a5afa33ba0775e85e8f144a4405b68 not found: ID does not exist"
Sep 18 00:02:54.870126 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.870080    2562 scope.go:117] "RemoveContainer" containerID="e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"
Sep 18 00:02:54.870304 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.870284    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68"} err="failed to get container status \"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68\": rpc error: code = NotFound desc = could not find container \"e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68\": container with ID starting with e539520956855bddd6bbf3b9e95761884ddfc8d8b457dd1200d464d962e70a68 not found: ID does not exist"
Sep 18 00:02:54.870388 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.870305    2562 scope.go:117] "RemoveContainer" containerID="460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"
Sep 18 00:02:54.870529 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.870514    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44"} err="failed to get container status \"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44\": rpc error: code = NotFound desc = could not find container \"460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44\": container with ID starting with 460c1ceb18ac9db70def259026099c05061f6d86b2cc43f066f841e5e3047d44 not found: ID does not exist"
Sep 18 00:02:54.870576 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.870529    2562 scope.go:117] "RemoveContainer" containerID="0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36"
Sep 18 00:02:54.878534 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.878511    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"]
Sep 18 00:02:54.879288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.879274    2562 scope.go:117] "RemoveContainer" containerID="d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6"
Sep 18 00:02:54.885268 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.885247    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-55w8zb4"]
Sep 18 00:02:54.888039 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.888026    2562 scope.go:117] "RemoveContainer" containerID="0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36"
Sep 18 00:02:54.888317 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:02:54.888298    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36\": container with ID starting with 0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36 not found: ID does not exist" containerID="0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36"
Sep 18 00:02:54.888366 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.888324    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36"} err="failed to get container status \"0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36\": rpc error: code = NotFound desc = could not find container \"0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36\": container with ID starting with 0bde32a6083af7d0bc7206d45727dbc9a114035e954010425df0b6af169b6c36 not found: ID does not exist"
Sep 18 00:02:54.888366 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.888341    2562 scope.go:117] "RemoveContainer" containerID="d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6"
Sep 18 00:02:54.888581 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:02:54.888561    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6\": container with ID starting with d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6 not found: ID does not exist" containerID="d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6"
Sep 18 00:02:54.888637 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:54.888590    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6"} err="failed to get container status \"d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6\": rpc error: code = NotFound desc = could not find container \"d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6\": container with ID starting with d42394f6c3d0c1d0fb18188c26d4fc30582c976feef4a1c63b9aac30c9a0dbf6 not found: ID does not exist"
Sep 18 00:02:55.418685 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:55.418609    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" path="/var/lib/kubelet/pods/056087a0-c8df-4f51-8561-7dc0324bcade/volumes"
Sep 18 00:02:55.419124 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:02:55.419110    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" path="/var/lib/kubelet/pods/9f8fe9c7-8688-4705-96b9-eb19d1b1e948/volumes"
Sep 18 00:03:26.505360 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505322    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"]
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505653    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="storage-initializer"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505664    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="storage-initializer"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505675    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="llm-d-routing-sidecar"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505681    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="llm-d-routing-sidecar"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505693    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505698    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505712    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="storage-initializer"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505717    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="storage-initializer"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505723    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505728    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505779    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="llm-d-routing-sidecar"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505787    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="056087a0-c8df-4f51-8561-7dc0324bcade" containerName="main"
Sep 18 00:03:26.505817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.505796    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f8fe9c7-8688-4705-96b9-eb19d1b1e948" containerName="main"
Sep 18 00:03:26.508887 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.508869    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.512246 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.512223    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"default-dockercfg-fm9k7\""
Sep 18 00:03:26.512404 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.512292    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"seaweedfs-s3-creds\""
Sep 18 00:03:26.512404 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.512363    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"lora-pvc-single-adapter-test-kserve-self-signed-certs\""
Sep 18 00:03:26.512794 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.512778    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:03:26.512885 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.512811    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"kube-root-ca.crt\""
Sep 18 00:03:26.515531 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.515264    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-4412bdb9\"/\"openshift-service-ca.crt\""
Sep 18 00:03:26.523964 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.523941    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"]
Sep 18 00:03:26.594423 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594390    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlttm\" (UniqueName: \"kubernetes.io/projected/d2b5ce0f-1524-4def-b2e7-99889537b15d-kube-api-access-jlttm\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.594602 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594435    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.594602 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594461    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b5ce0f-1524-4def-b2e7-99889537b15d-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.594602 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594497    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2b5ce0f-1524-4def-b2e7-99889537b15d-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.594602 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-home\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.594602 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594575    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.594779 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.594779 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.594662    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695242 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695208    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695242 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695254    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695501 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695297    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695501 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlttm\" (UniqueName: \"kubernetes.io/projected/d2b5ce0f-1524-4def-b2e7-99889537b15d-kube-api-access-jlttm\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695501 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695389    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695501 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695426    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b5ce0f-1524-4def-b2e7-99889537b15d-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695501 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695464    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2b5ce0f-1524-4def-b2e7-99889537b15d-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695501 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695491    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-home\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695833 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695740    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-model-cache\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695833 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695778    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-tmp-dir\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.695833 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.695813    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-home\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.696071 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.696048    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-kserve-provision-location\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.696387 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.696362    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2b5ce0f-1524-4def-b2e7-99889537b15d-cabundle-cert\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.697686 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.697660    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-dshm\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.698348 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.698321    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b5ce0f-1524-4def-b2e7-99889537b15d-tls-certs\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:26.704696 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:26.704672    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlttm\" (UniqueName: \"kubernetes.io/projected/d2b5ce0f-1524-4def-b2e7-99889537b15d-kube-api-access-jlttm\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.572829 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.572781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.578481 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.578452    2562 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.673590 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.673547    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.673795 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.673689    2562 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.676367 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.676343    2562 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") DevicePath \"csi-ec3744b588608b5db4687248d8c7427ed423304619420e48ce5f7757dec335b5\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.774198 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.774164    2562 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/116e18f5cf288dc8fecb0d77b30aea631f70117afbdf1ecc9da5a5d14780a735/globalmount\"" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.792711 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.792684    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") " pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:35.819743 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:35.819717    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:36.098437 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:36.098408    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"]
Sep 18 00:03:36.100279 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:03:36.100244    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b5ce0f_1524_4def_b2e7_99889537b15d.slice/crio-119a91b8e19c9367449aed1023ebd6a2aec2f3c1c69fadf61a98f2cacea875e3 WatchSource:0}: Error finding container 119a91b8e19c9367449aed1023ebd6a2aec2f3c1c69fadf61a98f2cacea875e3: Status 404 returned error can't find the container with id 119a91b8e19c9367449aed1023ebd6a2aec2f3c1c69fadf61a98f2cacea875e3
Sep 18 00:03:37.063177 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:37.063141    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" event={"ID":"d2b5ce0f-1524-4def-b2e7-99889537b15d","Type":"ContainerStarted","Data":"d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa"}
Sep 18 00:03:37.063662 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:37.063191    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" event={"ID":"d2b5ce0f-1524-4def-b2e7-99889537b15d","Type":"ContainerStarted","Data":"119a91b8e19c9367449aed1023ebd6a2aec2f3c1c69fadf61a98f2cacea875e3"}
Sep 18 00:03:39.076502 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:39.076465    2562 generic.go:356] "Generic (PLEG): container finished" podID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerID="d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa" exitCode=0
Sep 18 00:03:39.076888 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:39.076540    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" event={"ID":"d2b5ce0f-1524-4def-b2e7-99889537b15d","Type":"ContainerDied","Data":"d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa"}
Sep 18 00:03:40.083536 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:40.083497    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" event={"ID":"d2b5ce0f-1524-4def-b2e7-99889537b15d","Type":"ContainerStarted","Data":"7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee"}
Sep 18 00:03:40.107597 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:40.107552    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podStartSLOduration=11.776406734 podStartE2EDuration="14.107536388s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.331129654s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:03:40.104499307 +0000 UTC m=+3181.199026345" watchObservedRunningTime="2026-09-18 00:03:40.107536388 +0000 UTC m=+3181.202063464"
Sep 18 00:03:40.820389 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:40.820345    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:45.820852 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:45.820815    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:03:45.822494 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:45.822468    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:03:50.861787 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.861749    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"]
Sep 18 00:03:50.865985 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.865964    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.869289 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.869265    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 18 00:03:50.869289 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.869282    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Sep 18 00:03:50.869553 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.869528    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:03:50.869858 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.869842    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 18 00:03:50.869964 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.869861    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-nxglk\""
Sep 18 00:03:50.869964 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.869867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 18 00:03:50.878133 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.878109    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"]
Sep 18 00:03:50.904314 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904271    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.904472 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904332    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.904472 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.904472 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9bxq\" (UniqueName: \"kubernetes.io/projected/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kube-api-access-g9bxq\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.904598 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.904598 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904513    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.904598 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904539    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:50.904598 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:50.904570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.005808 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.005778    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006023 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.005817    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006023 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.005836    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006023 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.005855    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006023 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.005893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006023 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.005954    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006023 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.006017    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006356 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.006046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9bxq\" (UniqueName: \"kubernetes.io/projected/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kube-api-access-g9bxq\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006356 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.006220    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006356 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.006277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006511 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.006355    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.006511 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.006457    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.007022 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.006998    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.008404 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.008382    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.008738 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.008716    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.013619 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.013597    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9bxq\" (UniqueName: \"kubernetes.io/projected/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kube-api-access-g9bxq\") pod \"custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.180084 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.179985    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:03:51.460468 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:51.460445    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"]
Sep 18 00:03:51.463022 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:03:51.462995    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3f6191_ba6f_4174_aa8b_16e7de1c7b78.slice/crio-22cd90236e4ba27ba1b9ab72acce16b2dbad5e3f45746b81213200ea323aad41 WatchSource:0}: Error finding container 22cd90236e4ba27ba1b9ab72acce16b2dbad5e3f45746b81213200ea323aad41: Status 404 returned error can't find the container with id 22cd90236e4ba27ba1b9ab72acce16b2dbad5e3f45746b81213200ea323aad41
Sep 18 00:03:52.148977 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:52.148934    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" event={"ID":"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78","Type":"ContainerStarted","Data":"0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4"}
Sep 18 00:03:52.149449 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:52.149000    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" event={"ID":"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78","Type":"ContainerStarted","Data":"22cd90236e4ba27ba1b9ab72acce16b2dbad5e3f45746b81213200ea323aad41"}
Sep 18 00:03:54.164165 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:54.164076    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerID="0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4" exitCode=0
Sep 18 00:03:54.164165 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:54.164148    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" event={"ID":"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78","Type":"ContainerDied","Data":"0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4"}
Sep 18 00:03:55.173082 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:55.173044    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" event={"ID":"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78","Type":"ContainerStarted","Data":"e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8"}
Sep 18 00:03:55.199414 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:55.199341    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podStartSLOduration=2.838776141 podStartE2EDuration="5.199320947s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.360544806s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:03:55.19556805 +0000 UTC m=+3196.290095087" watchObservedRunningTime="2026-09-18 00:03:55.199320947 +0000 UTC m=+3196.293847984"
Sep 18 00:03:55.820357 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:03:55.820309    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:04:01.180189 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:01.180150    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:04:01.180820 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:01.180226    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:04:01.181599 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:01.181566    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:04:05.820851 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:05.820798    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:04:11.180999 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:11.180950    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:04:15.820836 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:15.820786    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:04:21.180626 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:21.180578    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:04:25.820223 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:25.820164    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:04:31.181336 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:31.181288    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:04:35.820351 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:35.820250    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:04:41.181260 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:41.181211    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:04:45.820624 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:45.820578    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:04:51.180829 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:51.180785    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:04:55.831421 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:55.831380    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:04:55.840196 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:04:55.840170    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:05:01.181380 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:01.181325    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:05:11.180971 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:11.180916    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:05:21.181007 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:21.180954    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:05:28.098072 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:28.098039    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4_d2b5ce0f-1524-4def-b2e7-99889537b15d/storage-initializer/0.log"
Sep 18 00:05:28.109607 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:28.109583    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-4412bdb9_lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4_d2b5ce0f-1524-4def-b2e7-99889537b15d/main/0.log"
Sep 18 00:05:29.294030 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:29.293995    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"]
Sep 18 00:05:31.181277 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:31.181231    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:05:41.180753 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:41.180707    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:05:44.316460 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:44.316382    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" containerID="cri-o://7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee" gracePeriod=45
Sep 18 00:05:44.832658 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:44.832612    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Sep 18 00:05:45.269700 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.269669    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:05:45.373292 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373259    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-tmp-dir\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373303    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-dshm\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373417    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"lora-pvc-lora-adapter-1\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373467    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-home\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373494    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-model-cache\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373536    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlttm\" (UniqueName: \"kubernetes.io/projected/d2b5ce0f-1524-4def-b2e7-99889537b15d-kube-api-access-jlttm\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373629    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-kserve-provision-location\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373663    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b5ce0f-1524-4def-b2e7-99889537b15d-tls-certs\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.373726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373691    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2b5ce0f-1524-4def-b2e7-99889537b15d-cabundle-cert\") pod \"d2b5ce0f-1524-4def-b2e7-99889537b15d\" (UID: \"d2b5ce0f-1524-4def-b2e7-99889537b15d\") "
Sep 18 00:05:45.374275 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.373880    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-home" (OuterVolumeSpecName: "home") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:05:45.374275 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.374252    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b5ce0f-1524-4def-b2e7-99889537b15d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:05:45.374603 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.374556    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:05:45.376156 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.376123    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b5ce0f-1524-4def-b2e7-99889537b15d-kube-api-access-jlttm" (OuterVolumeSpecName: "kube-api-access-jlttm") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "kube-api-access-jlttm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:05:45.376982 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.376958    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b5ce0f-1524-4def-b2e7-99889537b15d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:05:45.377616 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.377574    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-dshm" (OuterVolumeSpecName: "dshm") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:05:45.381084 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.381060    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47" (OuterVolumeSpecName: "lora-pvc-lora-adapter-1") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 00:05:45.388689 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.388664    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:05:45.400585 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.400543    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2b5ce0f-1524-4def-b2e7-99889537b15d" (UID: "d2b5ce0f-1524-4def-b2e7-99889537b15d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:05:45.474730 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474691    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.474730 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474719    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b5ce0f-1524-4def-b2e7-99889537b15d-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.474730 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474730    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2b5ce0f-1524-4def-b2e7-99889537b15d-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.475057 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474743    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.475057 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474751    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.475057 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474777    2562 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") on node \"ip-10-0-132-155.ec2.internal\" "
Sep 18 00:05:45.475057 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474790    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.475057 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474799    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2b5ce0f-1524-4def-b2e7-99889537b15d-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.475057 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.474808    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlttm\" (UniqueName: \"kubernetes.io/projected/d2b5ce0f-1524-4def-b2e7-99889537b15d-kube-api-access-jlttm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:05:45.485122 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.485099    2562 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47") on node "ip-10-0-132-155.ec2.internal"
Sep 18 00:05:45.576216 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.576171    2562 reconciler_common.go:299] "Volume detached for volume \"pvc-bda8df83-b1be-4d0a-9a69-400b8e8db424\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0931d347071494b47\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"csi-ec3744b588608b5db4687248d8c7427ed423304619420e48ce5f7757dec335b5\""
Sep 18 00:05:45.834502 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.834459    2562 generic.go:356] "Generic (PLEG): container finished" podID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerID="7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee" exitCode=0
Sep 18 00:05:45.834692 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.834518    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"
Sep 18 00:05:45.834692 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.834535    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" event={"ID":"d2b5ce0f-1524-4def-b2e7-99889537b15d","Type":"ContainerDied","Data":"7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee"}
Sep 18 00:05:45.834692 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.834589    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4" event={"ID":"d2b5ce0f-1524-4def-b2e7-99889537b15d","Type":"ContainerDied","Data":"119a91b8e19c9367449aed1023ebd6a2aec2f3c1c69fadf61a98f2cacea875e3"}
Sep 18 00:05:45.834692 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.834613    2562 scope.go:117] "RemoveContainer" containerID="7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee"
Sep 18 00:05:45.843423 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.843401    2562 scope.go:117] "RemoveContainer" containerID="d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa"
Sep 18 00:05:45.851397 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.851374    2562 scope.go:117] "RemoveContainer" containerID="7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee"
Sep 18 00:05:45.851683 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:05:45.851653    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee\": container with ID starting with 7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee not found: ID does not exist" containerID="7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee"
Sep 18 00:05:45.851798 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.851683    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee"} err="failed to get container status \"7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee\": rpc error: code = NotFound desc = could not find container \"7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee\": container with ID starting with 7270b5ee23837d1d3ca3e8a2b3b7255bc0d4ddaf127c55b18fa4207ecabd80ee not found: ID does not exist"
Sep 18 00:05:45.851798 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.851700    2562 scope.go:117] "RemoveContainer" containerID="d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa"
Sep 18 00:05:45.852052 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:05:45.852029    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa\": container with ID starting with d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa not found: ID does not exist" containerID="d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa"
Sep 18 00:05:45.852136 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.852055    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa"} err="failed to get container status \"d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa\": rpc error: code = NotFound desc = could not find container \"d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa\": container with ID starting with d4075a5a56100fd262bea1226b4e74912348b7eb13bbe1948be0c531227d01fa not found: ID does not exist"
Sep 18 00:05:45.855625 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.855595    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"]
Sep 18 00:05:45.860600 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:45.860567    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-4412bdb9/lora-pvc-single-adapter-test-kserve-6594f67c68-x7vw4"]
Sep 18 00:05:47.419820 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:47.419776    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" path="/var/lib/kubelet/pods/d2b5ce0f-1524-4def-b2e7-99889537b15d/volumes"
Sep 18 00:05:51.180694 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:51.180640    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:05:52.567105 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.567056    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"]
Sep 18 00:05:52.567665 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.567415    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="storage-initializer"
Sep 18 00:05:52.567665 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.567428    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="storage-initializer"
Sep 18 00:05:52.567665 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.567442    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main"
Sep 18 00:05:52.567665 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.567448    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main"
Sep 18 00:05:52.567665 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.567513    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b5ce0f-1524-4def-b2e7-99889537b15d" containerName="main"
Sep 18 00:05:52.570400 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.570379    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:52.573085 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.573062    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 00:05:52.573207 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.573068    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-fm5d8\""
Sep 18 00:05:52.574071 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.574051    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"seaweedfs-s3-creds\""
Sep 18 00:05:52.574210 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.574074    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 00:05:52.577390 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.577368    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"]
Sep 18 00:05:52.735014 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.734979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcmlp\" (UniqueName: \"kubernetes.io/projected/28ab56e5-ea7d-4d27-99c8-87ee5be8ac31-kube-api-access-vcmlp\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:52.835568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.835535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcmlp\" (UniqueName: \"kubernetes.io/projected/28ab56e5-ea7d-4d27-99c8-87ee5be8ac31-kube-api-access-vcmlp\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:52.843731 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:52.843692    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcmlp\" (UniqueName: \"kubernetes.io/projected/28ab56e5-ea7d-4d27-99c8-87ee5be8ac31-kube-api-access-vcmlp\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.488663 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.488626    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.494605 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.494574    2562 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.589211 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.589179    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.589380 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.589290    2562 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.593082 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.593062    2562 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") DevicePath \"csi-a0baf46cd2ed97419e9f70ef67c77ed44907d3156f6a79b00bc3b3ec9c08ba2b\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.802127 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.802085    2562 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b8114e6f1139c6eb6c3cd256b2938a9be9d75cd782455b6472a8dc4a10c54d97/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.815852 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.815824    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"e2e-pvc-lora-download-g2fsl\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:58.881325 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:58.881286    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:05:59.155622 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:59.155580    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"]
Sep 18 00:05:59.156699 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:05:59.156670    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28ab56e5_ea7d_4d27_99c8_87ee5be8ac31.slice/crio-1ca12502c88bf8f65451b6c88c4dd4ab3e51f5b13dd0fc3aa0374707840d89a2 WatchSource:0}: Error finding container 1ca12502c88bf8f65451b6c88c4dd4ab3e51f5b13dd0fc3aa0374707840d89a2: Status 404 returned error can't find the container with id 1ca12502c88bf8f65451b6c88c4dd4ab3e51f5b13dd0fc3aa0374707840d89a2
Sep 18 00:05:59.913106 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:59.913063    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl" event={"ID":"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31","Type":"ContainerStarted","Data":"37f1c704ede90c002d6a3f3c62adf20a3b12e18f5b2cfdfaaefee53d1306e3b3"}
Sep 18 00:05:59.913483 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:59.913115    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl" event={"ID":"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31","Type":"ContainerStarted","Data":"1ca12502c88bf8f65451b6c88c4dd4ab3e51f5b13dd0fc3aa0374707840d89a2"}
Sep 18 00:05:59.931160 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:05:59.931103    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl" podStartSLOduration=10.931087183 podStartE2EDuration="10.931087183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:05:59.929468844 +0000 UTC m=+3321.023995880" watchObservedRunningTime="2026-09-18 00:05:59.931087183 +0000 UTC m=+3321.025614219"
Sep 18 00:06:01.180671 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:01.180621    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Sep 18 00:06:01.926795 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:01.926761    2562 generic.go:356] "Generic (PLEG): container finished" podID="28ab56e5-ea7d-4d27-99c8-87ee5be8ac31" containerID="37f1c704ede90c002d6a3f3c62adf20a3b12e18f5b2cfdfaaefee53d1306e3b3" exitCode=0
Sep 18 00:06:01.926999 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:01.926826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl" event={"ID":"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31","Type":"ContainerDied","Data":"37f1c704ede90c002d6a3f3c62adf20a3b12e18f5b2cfdfaaefee53d1306e3b3"}
Sep 18 00:06:03.070569 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.070544    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:06:03.235013 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.234930    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcmlp\" (UniqueName: \"kubernetes.io/projected/28ab56e5-ea7d-4d27-99c8-87ee5be8ac31-kube-api-access-vcmlp\") pod \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") "
Sep 18 00:06:03.235159 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.235111    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\" (UID: \"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31\") "
Sep 18 00:06:03.237319 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.237282    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ab56e5-ea7d-4d27-99c8-87ee5be8ac31-kube-api-access-vcmlp" (OuterVolumeSpecName: "kube-api-access-vcmlp") pod "28ab56e5-ea7d-4d27-99c8-87ee5be8ac31" (UID: "28ab56e5-ea7d-4d27-99c8-87ee5be8ac31"). InnerVolumeSpecName "kube-api-access-vcmlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:06:03.237871 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.237850    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1" (OuterVolumeSpecName: "model-storage") pod "28ab56e5-ea7d-4d27-99c8-87ee5be8ac31" (UID: "28ab56e5-ea7d-4d27-99c8-87ee5be8ac31"). InnerVolumeSpecName "pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 00:06:03.336007 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.335974    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcmlp\" (UniqueName: \"kubernetes.io/projected/28ab56e5-ea7d-4d27-99c8-87ee5be8ac31-kube-api-access-vcmlp\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:03.336182 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.336027    2562 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") on node \"ip-10-0-132-155.ec2.internal\" "
Sep 18 00:06:03.938371 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.938341    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"
Sep 18 00:06:03.938580 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.938338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl" event={"ID":"28ab56e5-ea7d-4d27-99c8-87ee5be8ac31","Type":"ContainerDied","Data":"1ca12502c88bf8f65451b6c88c4dd4ab3e51f5b13dd0fc3aa0374707840d89a2"}
Sep 18 00:06:03.938580 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:03.938455    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ca12502c88bf8f65451b6c88c4dd4ab3e51f5b13dd0fc3aa0374707840d89a2"
Sep 18 00:06:04.310769 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:04.310743    2562 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1") on node "ip-10-0-132-155.ec2.internal"
Sep 18 00:06:04.343093 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:04.343055    2562 reconciler_common.go:299] "Volume detached for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"csi-a0baf46cd2ed97419e9f70ef67c77ed44907d3156f6a79b00bc3b3ec9c08ba2b\""
Sep 18 00:06:11.190790 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:11.190759    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:06:11.198785 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:11.198749    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:06:18.198429 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.198392    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"]
Sep 18 00:06:18.198821 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.198743    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ab56e5-ea7d-4d27-99c8-87ee5be8ac31" containerName="storage-initializer"
Sep 18 00:06:18.198821 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.198754    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ab56e5-ea7d-4d27-99c8-87ee5be8ac31" containerName="storage-initializer"
Sep 18 00:06:18.198821 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.198815    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ab56e5-ea7d-4d27-99c8-87ee5be8ac31" containerName="storage-initializer"
Sep 18 00:06:18.202087 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.202056    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.204896 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.204865    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Sep 18 00:06:18.205046 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.204912    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Sep 18 00:06:18.205046 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.204928    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-fm5d8\""
Sep 18 00:06:18.205046 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.204986    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Sep 18 00:06:18.211822 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.211792    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"]
Sep 18 00:06:18.263881 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.263841    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.264089 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.263895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.264089 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.263956    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.264089 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.264048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271e958e-fb73-4719-8857-653ac3eb98b9-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.264251 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.264149    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.264251 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.264187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qp8\" (UniqueName: \"kubernetes.io/projected/271e958e-fb73-4719-8857-653ac3eb98b9-kube-api-access-d4qp8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.365554 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.365495    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.365554 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.365557    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qp8\" (UniqueName: \"kubernetes.io/projected/271e958e-fb73-4719-8857-653ac3eb98b9-kube-api-access-d4qp8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.365823 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.365664    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.365823 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.365702    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.365823 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.365722    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.365823 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.365767    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271e958e-fb73-4719-8857-653ac3eb98b9-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.366075 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.366056    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-model-cache\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.366123 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.366073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-home\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.366161 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.366126    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.367944 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.367920    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-dshm\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.368225 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.368209    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271e958e-fb73-4719-8857-653ac3eb98b9-tls-certs\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:18.373098 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:18.373077    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qp8\" (UniqueName: \"kubernetes.io/projected/271e958e-fb73-4719-8857-653ac3eb98b9-kube-api-access-d4qp8\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.584955 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.584890    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.590509 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.590482    2562 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.685541 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.685507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.685735 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.685659    2562 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.688373 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.688351    2562 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") DevicePath \"csi-a0baf46cd2ed97419e9f70ef67c77ed44907d3156f6a79b00bc3b3ec9c08ba2b\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.776784 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.776746    2562 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b8114e6f1139c6eb6c3cd256b2938a9be9d75cd782455b6472a8dc4a10c54d97/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.796583 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.796550    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:20.912689 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:20.912584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:21.191174 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:21.191144    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"]
Sep 18 00:06:21.193463 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:06:21.193438    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271e958e_fb73_4719_8857_653ac3eb98b9.slice/crio-93984beda7e5920a31bab133ddc39052411c395485d371ae8371d50326552089 WatchSource:0}: Error finding container 93984beda7e5920a31bab133ddc39052411c395485d371ae8371d50326552089: Status 404 returned error can't find the container with id 93984beda7e5920a31bab133ddc39052411c395485d371ae8371d50326552089
Sep 18 00:06:22.035806 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:22.035764    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" event={"ID":"271e958e-fb73-4719-8857-653ac3eb98b9","Type":"ContainerStarted","Data":"f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26"}
Sep 18 00:06:22.036314 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:22.035829    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" event={"ID":"271e958e-fb73-4719-8857-653ac3eb98b9","Type":"ContainerStarted","Data":"93984beda7e5920a31bab133ddc39052411c395485d371ae8371d50326552089"}
Sep 18 00:06:22.057218 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:22.057155    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podStartSLOduration=4.057136177 podStartE2EDuration="4.057136177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:06:22.054419265 +0000 UTC m=+3343.148946301" watchObservedRunningTime="2026-09-18 00:06:22.057136177 +0000 UTC m=+3343.151663213"
Sep 18 00:06:22.912932 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:22.912862    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:30.913560 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:30.913527    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:06:30.915372 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:30.915343    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:06:34.651048 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:34.651003    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"]
Sep 18 00:06:40.913291 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:40.913247    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:06:49.674792 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:49.674714    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main" containerID="cri-o://e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8" gracePeriod=45
Sep 18 00:06:50.435644 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.435616    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:06:50.540969 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.540878    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kserve-provision-location\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.540969 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.540927    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-model-cache\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.540969 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.540947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-cabundle-cert\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.541259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.540980    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-dshm\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.541259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541034    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tmp-dir\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.541259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9bxq\" (UniqueName: \"kubernetes.io/projected/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kube-api-access-g9bxq\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.541259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541095    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tls-certs\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.541259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541116    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-home\") pod \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\" (UID: \"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78\") "
Sep 18 00:06:50.541259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541237    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.541574 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541393    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.541726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541700    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:06:50.541985 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.541956    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-home" (OuterVolumeSpecName: "home") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.543960 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.543929    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-dshm" (OuterVolumeSpecName: "dshm") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.544276 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.544245    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kube-api-access-g9bxq" (OuterVolumeSpecName: "kube-api-access-g9bxq") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "kube-api-access-g9bxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:06:50.544360 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.544266    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:06:50.553536 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.553452    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.571112 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.571068    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" (UID: "cd3f6191-ba6f-4174-aa8b-16e7de1c7b78"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:06:50.642202 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.642164    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.642202 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.642200    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9bxq\" (UniqueName: \"kubernetes.io/projected/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kube-api-access-g9bxq\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.642405 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.642211    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.642405 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.642222    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.642405 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.642230    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.642405 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.642240    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.642405 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.642248    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:06:50.913383 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:50.913333    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:06:51.207469 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.207371    2562 generic.go:356] "Generic (PLEG): container finished" podID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerID="e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8" exitCode=0
Sep 18 00:06:51.207469 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.207421    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"
Sep 18 00:06:51.207469 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.207433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" event={"ID":"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78","Type":"ContainerDied","Data":"e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8"}
Sep 18 00:06:51.207738 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.207486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8" event={"ID":"cd3f6191-ba6f-4174-aa8b-16e7de1c7b78","Type":"ContainerDied","Data":"22cd90236e4ba27ba1b9ab72acce16b2dbad5e3f45746b81213200ea323aad41"}
Sep 18 00:06:51.207738 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.207515    2562 scope.go:117] "RemoveContainer" containerID="e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8"
Sep 18 00:06:51.220753 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.220722    2562 scope.go:117] "RemoveContainer" containerID="0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4"
Sep 18 00:06:51.231528 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.231499    2562 scope.go:117] "RemoveContainer" containerID="e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8"
Sep 18 00:06:51.232068 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:06:51.232007    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8\": container with ID starting with e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8 not found: ID does not exist" containerID="e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8"
Sep 18 00:06:51.232183 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.232054    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8"} err="failed to get container status \"e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8\": rpc error: code = NotFound desc = could not find container \"e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8\": container with ID starting with e21b731148ddab56b6fc3148c06698944f99256a8f9ca7a46e1477201c9eb0a8 not found: ID does not exist"
Sep 18 00:06:51.232183 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.232086    2562 scope.go:117] "RemoveContainer" containerID="0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4"
Sep 18 00:06:51.232718 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:06:51.232578    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4\": container with ID starting with 0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4 not found: ID does not exist" containerID="0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4"
Sep 18 00:06:51.232718 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.232617    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4"} err="failed to get container status \"0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4\": rpc error: code = NotFound desc = could not find container \"0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4\": container with ID starting with 0c73ab629517bdb45d1f9e20e0b68d8dad56aa8940566b1ae1a130f35affe1a4 not found: ID does not exist"
Sep 18 00:06:51.235310 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.235280    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"]
Sep 18 00:06:51.241833 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.241799    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-prefill-69d5bbdc57-6cbb8"]
Sep 18 00:06:51.418986 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:06:51.418954    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" path="/var/lib/kubelet/pods/cd3f6191-ba6f-4174-aa8b-16e7de1c7b78/volumes"
Sep 18 00:07:00.913585 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:00.913536    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:07:10.913779 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:10.913738    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:07:20.913936 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:20.913869    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:07:30.914068 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:30.914013    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:07:37.897940 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.897885    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"]
Sep 18 00:07:37.898456 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.898435    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="storage-initializer"
Sep 18 00:07:37.898533 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.898459    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="storage-initializer"
Sep 18 00:07:37.898533 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.898496    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main"
Sep 18 00:07:37.898533 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.898506    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main"
Sep 18 00:07:37.898704 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.898610    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3f6191-ba6f-4174-aa8b-16e7de1c7b78" containerName="main"
Sep 18 00:07:37.902241 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.902218    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.905471 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.905447    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Sep 18 00:07:37.905600 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.905570    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 18 00:07:37.905983 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.905947    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-gxpf2\""
Sep 18 00:07:37.906083 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.905953    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:07:37.906363 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.906350    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 18 00:07:37.906629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.906611    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 18 00:07:37.927259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.927231    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"]
Sep 18 00:07:37.967769 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.967728    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.967969 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.967772    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950195c2-e4bf-4097-8002-9860aee1aaa9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.967969 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.967823    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.967969 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.967861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/950195c2-e4bf-4097-8002-9860aee1aaa9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.967969 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.967938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-home\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.968189 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.967993    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/950195c2-e4bf-4097-8002-9860aee1aaa9-kube-api-access-crgrg\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.968189 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.968025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:37.968189 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:37.968084    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.068831 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.068792    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.068831 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.068832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950195c2-e4bf-4097-8002-9860aee1aaa9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069108 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.068853    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069108 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.068946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/950195c2-e4bf-4097-8002-9860aee1aaa9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069108 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069085    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-home\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069279 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069144    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/950195c2-e4bf-4097-8002-9860aee1aaa9-kube-api-access-crgrg\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069279 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069188    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069279 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069234    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069436 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069320    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069436 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069338    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069735 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069709    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/950195c2-e4bf-4097-8002-9860aee1aaa9-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069848 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069782    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.069848 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.069799    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-home\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.071458 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.071430    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.071458 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.071440    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950195c2-e4bf-4097-8002-9860aee1aaa9-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.077245 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.077222    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/950195c2-e4bf-4097-8002-9860aee1aaa9-kube-api-access-crgrg\") pod \"router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.216416 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.216322    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:38.495926 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:38.495883    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"]
Sep 18 00:07:38.497370 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:07:38.497343    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950195c2_e4bf_4097_8002_9860aee1aaa9.slice/crio-6ebe627ab233dce1980d538fa4cd195fd4b29a679d9908c6760f9f30b69b7f6d WatchSource:0}: Error finding container 6ebe627ab233dce1980d538fa4cd195fd4b29a679d9908c6760f9f30b69b7f6d: Status 404 returned error can't find the container with id 6ebe627ab233dce1980d538fa4cd195fd4b29a679d9908c6760f9f30b69b7f6d
Sep 18 00:07:39.483058 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:39.483008    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" event={"ID":"950195c2-e4bf-4097-8002-9860aee1aaa9","Type":"ContainerStarted","Data":"a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b"}
Sep 18 00:07:39.483611 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:39.483089    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" event={"ID":"950195c2-e4bf-4097-8002-9860aee1aaa9","Type":"ContainerStarted","Data":"6ebe627ab233dce1980d538fa4cd195fd4b29a679d9908c6760f9f30b69b7f6d"}
Sep 18 00:07:40.923634 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:40.923602    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:07:40.931489 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:40.931462    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:07:41.494888 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:41.494844    2562 generic.go:356] "Generic (PLEG): container finished" podID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerID="a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b" exitCode=0
Sep 18 00:07:41.495084 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:41.494926    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" event={"ID":"950195c2-e4bf-4097-8002-9860aee1aaa9","Type":"ContainerDied","Data":"a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b"}
Sep 18 00:07:42.501861 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:42.501824    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" event={"ID":"950195c2-e4bf-4097-8002-9860aee1aaa9","Type":"ContainerStarted","Data":"85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693"}
Sep 18 00:07:42.524742 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:42.524687    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podStartSLOduration=3.309740707 podStartE2EDuration="5.524669728s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.214929021s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:07:42.521867699 +0000 UTC m=+3423.616394735" watchObservedRunningTime="2026-09-18 00:07:42.524669728 +0000 UTC m=+3423.619196764"
Sep 18 00:07:48.217128 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:48.217084    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:48.217544 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:48.217152    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:07:48.218494 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:48.218459    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:07:58.217320 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:07:58.217264    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:08:08.217298 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:08.217245    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:08:18.217465 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:18.217420    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:08:18.711193 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:18.711151    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm_271e958e-fb73-4719-8857-653ac3eb98b9/main/0.log"
Sep 18 00:08:19.612054 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:19.612017    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"]
Sep 18 00:08:28.217710 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:28.217667    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:08:34.640083 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:34.639999    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" containerID="cri-o://f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26" gracePeriod=45
Sep 18 00:08:34.924582 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:34.924483    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Sep 18 00:08:35.397060 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.397030    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:08:35.509498 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509395    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-home\") pod \"271e958e-fb73-4719-8857-653ac3eb98b9\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") "
Sep 18 00:08:35.509498 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509450    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-dshm\") pod \"271e958e-fb73-4719-8857-653ac3eb98b9\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") "
Sep 18 00:08:35.509498 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509469    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4qp8\" (UniqueName: \"kubernetes.io/projected/271e958e-fb73-4719-8857-653ac3eb98b9-kube-api-access-d4qp8\") pod \"271e958e-fb73-4719-8857-653ac3eb98b9\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") "
Sep 18 00:08:35.509498 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509503    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-tmp-dir\") pod \"271e958e-fb73-4719-8857-653ac3eb98b9\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") "
Sep 18 00:08:35.509847 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509608    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") pod \"271e958e-fb73-4719-8857-653ac3eb98b9\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") "
Sep 18 00:08:35.509847 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509639    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-model-cache\") pod \"271e958e-fb73-4719-8857-653ac3eb98b9\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") "
Sep 18 00:08:35.509847 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509715    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271e958e-fb73-4719-8857-653ac3eb98b9-tls-certs\") pod \"271e958e-fb73-4719-8857-653ac3eb98b9\" (UID: \"271e958e-fb73-4719-8857-653ac3eb98b9\") "
Sep 18 00:08:35.510063 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.509873    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-home" (OuterVolumeSpecName: "home") pod "271e958e-fb73-4719-8857-653ac3eb98b9" (UID: "271e958e-fb73-4719-8857-653ac3eb98b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:08:35.510119 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.510092    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "271e958e-fb73-4719-8857-653ac3eb98b9" (UID: "271e958e-fb73-4719-8857-653ac3eb98b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:08:35.510422 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.510213    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:08:35.510422 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.510238    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:08:35.511934 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.511890    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271e958e-fb73-4719-8857-653ac3eb98b9-kube-api-access-d4qp8" (OuterVolumeSpecName: "kube-api-access-d4qp8") pod "271e958e-fb73-4719-8857-653ac3eb98b9" (UID: "271e958e-fb73-4719-8857-653ac3eb98b9"). InnerVolumeSpecName "kube-api-access-d4qp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:08:35.512680 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.512633    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-dshm" (OuterVolumeSpecName: "dshm") pod "271e958e-fb73-4719-8857-653ac3eb98b9" (UID: "271e958e-fb73-4719-8857-653ac3eb98b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:08:35.512680 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.512655    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/271e958e-fb73-4719-8857-653ac3eb98b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "271e958e-fb73-4719-8857-653ac3eb98b9" (UID: "271e958e-fb73-4719-8857-653ac3eb98b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:08:35.517563 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.517539    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1" (OuterVolumeSpecName: "kserve-pvc-source") pod "271e958e-fb73-4719-8857-653ac3eb98b9" (UID: "271e958e-fb73-4719-8857-653ac3eb98b9"). InnerVolumeSpecName "pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Sep 18 00:08:35.521796 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.521774    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "271e958e-fb73-4719-8857-653ac3eb98b9" (UID: "271e958e-fb73-4719-8857-653ac3eb98b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:08:35.610782 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.610744    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:08:35.611006 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.610802    2562 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") on node \"ip-10-0-132-155.ec2.internal\" "
Sep 18 00:08:35.611006 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.610817    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/271e958e-fb73-4719-8857-653ac3eb98b9-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:08:35.611006 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.610831    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/271e958e-fb73-4719-8857-653ac3eb98b9-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:08:35.611006 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.610843    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qp8\" (UniqueName: \"kubernetes.io/projected/271e958e-fb73-4719-8857-653ac3eb98b9-kube-api-access-d4qp8\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:08:35.642086 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.642060    2562 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1") on node "ip-10-0-132-155.ec2.internal"
Sep 18 00:08:35.711747 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.711709    2562 reconciler_common.go:299] "Volume detached for volume \"pvc-6c7dd4d3-8a33-4fbd-9173-946c522dbf05\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aacb9975c521dcf1\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"csi-a0baf46cd2ed97419e9f70ef67c77ed44907d3156f6a79b00bc3b3ec9c08ba2b\""
Sep 18 00:08:35.822325 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.822235    2562 generic.go:356] "Generic (PLEG): container finished" podID="271e958e-fb73-4719-8857-653ac3eb98b9" containerID="f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26" exitCode=0
Sep 18 00:08:35.822325 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.822288    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"
Sep 18 00:08:35.822568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.822323    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" event={"ID":"271e958e-fb73-4719-8857-653ac3eb98b9","Type":"ContainerDied","Data":"f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26"}
Sep 18 00:08:35.822568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.822385    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm" event={"ID":"271e958e-fb73-4719-8857-653ac3eb98b9","Type":"ContainerDied","Data":"93984beda7e5920a31bab133ddc39052411c395485d371ae8371d50326552089"}
Sep 18 00:08:35.822568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.822413    2562 scope.go:117] "RemoveContainer" containerID="f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26"
Sep 18 00:08:35.832444 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.832423    2562 scope.go:117] "RemoveContainer" containerID="f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26"
Sep 18 00:08:35.832709 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:08:35.832691    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26\": container with ID starting with f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26 not found: ID does not exist" containerID="f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26"
Sep 18 00:08:35.832772 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.832716    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26"} err="failed to get container status \"f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26\": rpc error: code = NotFound desc = could not find container \"f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26\": container with ID starting with f13593ea0790f1dc0c5951a9f73f16cfe99d9affbdc2100bc1849d60f8017c26 not found: ID does not exist"
Sep 18 00:08:35.844477 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.844454    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"]
Sep 18 00:08:35.849810 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:35.849788    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-8cfd5947f-sgdgm"]
Sep 18 00:08:37.419100 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:37.419064    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" path="/var/lib/kubelet/pods/271e958e-fb73-4719-8857-653ac3eb98b9/volumes"
Sep 18 00:08:38.217738 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:38.217689    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:08:44.824064 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:44.824029    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"]
Sep 18 00:08:44.827352 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:44.827326    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/e2e-pvc-lora-download-g2fsl"]
Sep 18 00:08:45.420051 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:45.420009    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ab56e5-ea7d-4d27-99c8-87ee5be8ac31" path="/var/lib/kubelet/pods/28ab56e5-ea7d-4d27-99c8-87ee5be8ac31/volumes"
Sep 18 00:08:48.216961 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:48.216882    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:08:49.210806 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.210766    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"]
Sep 18 00:08:49.211225 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.211205    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main"
Sep 18 00:08:49.211225 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.211227    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main"
Sep 18 00:08:49.211392 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.211289    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="271e958e-fb73-4719-8857-653ac3eb98b9" containerName="main"
Sep 18 00:08:49.216423 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.216400    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.219147 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.219112    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"seaweedfs-s3-creds\""
Sep 18 00:08:49.219510 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.219151    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 18 00:08:49.219510 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.219162    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-s5dlv\""
Sep 18 00:08:49.219510 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.219394    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:08:49.220179 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.220164    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 18 00:08:49.220240 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.220185    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 18 00:08:49.229425 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.229402    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"]
Sep 18 00:08:49.330980 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.330932    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-home\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.331162 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.330997    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnmfs\" (UniqueName: \"kubernetes.io/projected/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kube-api-access-mnmfs\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.331162 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.331034    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tls-certs\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.331162 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.331114    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tmp-dir\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.331315 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.331164    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-model-cache\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.331315 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.331228    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-dshm\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.331315 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.331281    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kserve-provision-location\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.331416 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.331326    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-cabundle-cert\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432417 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432384    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-cabundle-cert\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432612 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432440    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-home\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432612 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432457    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnmfs\" (UniqueName: \"kubernetes.io/projected/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kube-api-access-mnmfs\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432612 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432575    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tls-certs\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432793 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432643    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tmp-dir\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432793 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-model-cache\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432793 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432709    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-dshm\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.432793 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.432745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kserve-provision-location\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.433076 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.433014    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tmp-dir\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.433214 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.433073    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-model-cache\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.433214 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.433100    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kserve-provision-location\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.433214 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.433119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-home\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.433409 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.433273    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-cabundle-cert\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.434948 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.434929    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-dshm\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.435387 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.435367    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tls-certs\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.441494 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.441469    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnmfs\" (UniqueName: \"kubernetes.io/projected/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kube-api-access-mnmfs\") pod \"tls-verification-test-kserve-7bdfd879b4-k5hh9\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.528075 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.527987    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:49.811333 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.811297    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"]
Sep 18 00:08:49.812568 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:08:49.812542    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad0da88_94d5_4f93_a74b_d1d7ae84ba57.slice/crio-d5a6bd0da84670188e6e94444403d38fc694846b53383981a56bd5397e092473 WatchSource:0}: Error finding container d5a6bd0da84670188e6e94444403d38fc694846b53383981a56bd5397e092473: Status 404 returned error can't find the container with id d5a6bd0da84670188e6e94444403d38fc694846b53383981a56bd5397e092473
Sep 18 00:08:49.908758 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.908722    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" event={"ID":"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57","Type":"ContainerStarted","Data":"607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b"}
Sep 18 00:08:49.909008 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:49.908777    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" event={"ID":"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57","Type":"ContainerStarted","Data":"d5a6bd0da84670188e6e94444403d38fc694846b53383981a56bd5397e092473"}
Sep 18 00:08:52.925990 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:52.925946    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerID="607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b" exitCode=0
Sep 18 00:08:52.926430 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:52.926014    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" event={"ID":"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57","Type":"ContainerDied","Data":"607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b"}
Sep 18 00:08:53.934038 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:53.933994    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" event={"ID":"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57","Type":"ContainerStarted","Data":"dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03"}
Sep 18 00:08:53.957224 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:53.957176    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podStartSLOduration=2.584414448 podStartE2EDuration="4.957151603s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.372737155s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:08:53.955855388 +0000 UTC m=+3495.050382424" watchObservedRunningTime="2026-09-18 00:08:53.957151603 +0000 UTC m=+3495.051678642"
Sep 18 00:08:54.528346 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:54.528300    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:58.217265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:58.217215    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:08:59.528661 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:59.528613    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:08:59.530276 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:08:59.530242    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:09:08.217506 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:08.217452    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:09:09.529134 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:09.529090    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:09:18.217625 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:18.217579    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:09:19.529407 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:19.529346    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:09:28.217682 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:28.217632    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": dial tcp 10.133.0.63:8000: connect: connection refused"
Sep 18 00:09:29.528911 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:29.528860    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:09:38.226874 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:38.226839    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:09:38.235384 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:38.235350    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:09:39.529365 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:39.529317    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:09:47.060727 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:47.060688    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"]
Sep 18 00:09:49.528995 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:49.528929    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:09:59.528757 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:09:59.528707    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:10:02.084964 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.084863    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main" containerID="cri-o://85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693" gracePeriod=45
Sep 18 00:10:02.854514 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.854479    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:10:02.890376 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890323    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950195c2-e4bf-4097-8002-9860aee1aaa9-tls-certs\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.890376 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890376    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-dshm\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.890640 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890405    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-model-cache\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.890640 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890457    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-home\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.890640 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890483    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/950195c2-e4bf-4097-8002-9860aee1aaa9-kube-api-access-crgrg\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.890640 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890551    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/950195c2-e4bf-4097-8002-9860aee1aaa9-cabundle-cert\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.890640 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890581    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-tmp-dir\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.890640 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890617    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-kserve-provision-location\") pod \"950195c2-e4bf-4097-8002-9860aee1aaa9\" (UID: \"950195c2-e4bf-4097-8002-9860aee1aaa9\") "
Sep 18 00:10:02.891059 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890722    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-model-cache" (OuterVolumeSpecName: "model-cache") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:02.891059 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.890994    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:02.891059 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.891036    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950195c2-e4bf-4097-8002-9860aee1aaa9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:10:02.891360 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.891323    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-home" (OuterVolumeSpecName: "home") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:02.893421 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.893340    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/950195c2-e4bf-4097-8002-9860aee1aaa9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:10:02.893583 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.893560    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950195c2-e4bf-4097-8002-9860aee1aaa9-kube-api-access-crgrg" (OuterVolumeSpecName: "kube-api-access-crgrg") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "kube-api-access-crgrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:10:02.894383 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.894363    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-dshm" (OuterVolumeSpecName: "dshm") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:02.909710 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.909658    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:02.918455 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.918415    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "950195c2-e4bf-4097-8002-9860aee1aaa9" (UID: "950195c2-e4bf-4097-8002-9860aee1aaa9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:10:02.991614 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.991570    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:02.991614 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.991607    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/950195c2-e4bf-4097-8002-9860aee1aaa9-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:02.991614 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.991618    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:02.991614 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.991627    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:02.991895 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.991641    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgrg\" (UniqueName: \"kubernetes.io/projected/950195c2-e4bf-4097-8002-9860aee1aaa9-kube-api-access-crgrg\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:02.991895 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.991665    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/950195c2-e4bf-4097-8002-9860aee1aaa9-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:02.991895 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:02.991675    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/950195c2-e4bf-4097-8002-9860aee1aaa9-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:10:03.344553 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.344519    2562 generic.go:356] "Generic (PLEG): container finished" podID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerID="85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693" exitCode=0
Sep 18 00:10:03.345020 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.344578    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"
Sep 18 00:10:03.345020 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.344578    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" event={"ID":"950195c2-e4bf-4097-8002-9860aee1aaa9","Type":"ContainerDied","Data":"85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693"}
Sep 18 00:10:03.345020 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.344687    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns" event={"ID":"950195c2-e4bf-4097-8002-9860aee1aaa9","Type":"ContainerDied","Data":"6ebe627ab233dce1980d538fa4cd195fd4b29a679d9908c6760f9f30b69b7f6d"}
Sep 18 00:10:03.345020 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.344719    2562 scope.go:117] "RemoveContainer" containerID="85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693"
Sep 18 00:10:03.356207 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.356178    2562 scope.go:117] "RemoveContainer" containerID="a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b"
Sep 18 00:10:03.365470 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.365346    2562 scope.go:117] "RemoveContainer" containerID="85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693"
Sep 18 00:10:03.365755 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:10:03.365725    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693\": container with ID starting with 85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693 not found: ID does not exist" containerID="85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693"
Sep 18 00:10:03.365876 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.365767    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693"} err="failed to get container status \"85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693\": rpc error: code = NotFound desc = could not find container \"85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693\": container with ID starting with 85edb5d0b0c2774fef5f263dfa5dc5c383ecf2ad59cf9edb9fae0b6400c87693 not found: ID does not exist"
Sep 18 00:10:03.365876 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.365794    2562 scope.go:117] "RemoveContainer" containerID="a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b"
Sep 18 00:10:03.366182 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:10:03.366161    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b\": container with ID starting with a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b not found: ID does not exist" containerID="a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b"
Sep 18 00:10:03.366265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.366192    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b"} err="failed to get container status \"a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b\": rpc error: code = NotFound desc = could not find container \"a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b\": container with ID starting with a1ec1f7ddad4d4ef6bb4425c95dc52c5a9861c4101af28f5917521ae164cf49b not found: ID does not exist"
Sep 18 00:10:03.367748 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.367723    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"]
Sep 18 00:10:03.371004 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.370979    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7486fd9995-2fxns"]
Sep 18 00:10:03.419963 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:03.419921    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" path="/var/lib/kubelet/pods/950195c2-e4bf-4097-8002-9860aee1aaa9/volumes"
Sep 18 00:10:09.529342 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:09.529286    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:10:19.529356 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:19.529312    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:10:29.529318 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:29.529253    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:10:39.528704 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:39.528656    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:10:49.539171 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:49.539131    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:10:49.551383 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:10:49.551342    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:11:41.134046 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:41.134009    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"]
Sep 18 00:11:56.156503 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.156454    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" containerID="cri-o://dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03" gracePeriod=45
Sep 18 00:11:56.540719 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.540618    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Sep 18 00:11:56.831668 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.831639    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:11:56.905628 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905597    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-model-cache\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.905628 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905633    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-dshm\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.905862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905654    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-home\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.905862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905682    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tmp-dir\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.905862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905720    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnmfs\" (UniqueName: \"kubernetes.io/projected/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kube-api-access-mnmfs\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.905862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905742    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kserve-provision-location\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.905862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905766    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tls-certs\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.905862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905795    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-cabundle-cert\") pod \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\" (UID: \"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57\") "
Sep 18 00:11:56.906215 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.905969    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:11:56.906387 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.906357    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:11:56.906596 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.906576    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-home" (OuterVolumeSpecName: "home") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:11:56.908096 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.908068    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kube-api-access-mnmfs" (OuterVolumeSpecName: "kube-api-access-mnmfs") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "kube-api-access-mnmfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:11:56.908354 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.908327    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:11:56.908354 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.908345    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-dshm" (OuterVolumeSpecName: "dshm") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:11:56.927036 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.927001    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:11:56.936374 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.936342    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" (UID: "8ad0da88-94d5-4f93-a74b-d1d7ae84ba57"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:11:56.964874 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.964842    2562 generic.go:356] "Generic (PLEG): container finished" podID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerID="dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03" exitCode=0
Sep 18 00:11:56.965044 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.964890    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"
Sep 18 00:11:56.965044 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.964924    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" event={"ID":"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57","Type":"ContainerDied","Data":"dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03"}
Sep 18 00:11:56.965044 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.964988    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9" event={"ID":"8ad0da88-94d5-4f93-a74b-d1d7ae84ba57","Type":"ContainerDied","Data":"d5a6bd0da84670188e6e94444403d38fc694846b53383981a56bd5397e092473"}
Sep 18 00:11:56.965044 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.965013    2562 scope.go:117] "RemoveContainer" containerID="dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03"
Sep 18 00:11:56.974874 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.974858    2562 scope.go:117] "RemoveContainer" containerID="607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b"
Sep 18 00:11:56.983367 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.983348    2562 scope.go:117] "RemoveContainer" containerID="dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03"
Sep 18 00:11:56.983620 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:11:56.983602    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03\": container with ID starting with dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03 not found: ID does not exist" containerID="dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03"
Sep 18 00:11:56.983684 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.983628    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03"} err="failed to get container status \"dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03\": rpc error: code = NotFound desc = could not find container \"dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03\": container with ID starting with dcd54b7b6e6c48b89c99cbe35718cdf8d71152595d5abaceb1f42844b8fbee03 not found: ID does not exist"
Sep 18 00:11:56.983684 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.983644    2562 scope.go:117] "RemoveContainer" containerID="607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b"
Sep 18 00:11:56.983888 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:11:56.983873    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b\": container with ID starting with 607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b not found: ID does not exist" containerID="607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b"
Sep 18 00:11:56.984020 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.983893    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b"} err="failed to get container status \"607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b\": rpc error: code = NotFound desc = could not find container \"607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b\": container with ID starting with 607b1e60e2d0afdeeff55e1b9bec5c074e3ef218ec6598243d40ea1a7168a82b not found: ID does not exist"
Sep 18 00:11:56.988133 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.988110    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"]
Sep 18 00:11:56.994304 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:56.994273    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-7bdfd879b4-k5hh9"]
Sep 18 00:11:57.006453 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006415    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.006453 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006446    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.006629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006485    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.006629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006499    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnmfs\" (UniqueName: \"kubernetes.io/projected/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kube-api-access-mnmfs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.006629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006513    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.006629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006525    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.006629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006538    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.006629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.006550    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:11:57.418283 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:11:57.418250    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" path="/var/lib/kubelet/pods/8ad0da88-94d5-4f93-a74b-d1d7ae84ba57/volumes"
Sep 18 00:12:39.117456 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117422    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"]
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117757    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main"
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117769    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main"
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117782    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main"
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117787    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main"
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117799    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="storage-initializer"
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117804    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="storage-initializer"
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117820    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="storage-initializer"
Sep 18 00:12:39.117868 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117826    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="storage-initializer"
Sep 18 00:12:39.118144 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117873    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ad0da88-94d5-4f93-a74b-d1d7ae84ba57" containerName="main"
Sep 18 00:12:39.118144 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.117885    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="950195c2-e4bf-4097-8002-9860aee1aaa9" containerName="main"
Sep 18 00:12:39.121145 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.121128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.124287 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.124262    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 18 00:12:39.124287 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.124277    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 18 00:12:39.124492 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.124269    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 00:12:39.124492 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.124325    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:12:39.124492 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.124341    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-vn5mx\""
Sep 18 00:12:39.125343 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.125327    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 18 00:12:39.132278 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.132258    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"]
Sep 18 00:12:39.161966 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.161919    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-home\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.161966 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.161967    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.162238 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.162056    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d85e89-5652-4427-92a3-2d74f1d28610-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.162238 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.162098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-dshm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.162238 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.162133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-tmp-dir\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.162238 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.162166    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-model-cache\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.162238 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.162200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d85e89-5652-4427-92a3-2d74f1d28610-tls-certs\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.162238 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.162233    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcnm\" (UniqueName: \"kubernetes.io/projected/b1d85e89-5652-4427-92a3-2d74f1d28610-kube-api-access-9bcnm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263583 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-home\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263583 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263589    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263856 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263622    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d85e89-5652-4427-92a3-2d74f1d28610-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263856 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263639    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-dshm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263856 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263659    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-tmp-dir\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263856 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263695    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-model-cache\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263856 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263718    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d85e89-5652-4427-92a3-2d74f1d28610-tls-certs\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.263856 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.263739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcnm\" (UniqueName: \"kubernetes.io/projected/b1d85e89-5652-4427-92a3-2d74f1d28610-kube-api-access-9bcnm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.264229 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.264066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-home\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.264229 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.264086    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.264229 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.264140    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-tmp-dir\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.264385 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.264369    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-model-cache\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.264452 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.264425    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d85e89-5652-4427-92a3-2d74f1d28610-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.266111 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.266084    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-dshm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.266275 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.266257    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d85e89-5652-4427-92a3-2d74f1d28610-tls-certs\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.271810 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.271783    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcnm\" (UniqueName: \"kubernetes.io/projected/b1d85e89-5652-4427-92a3-2d74f1d28610-kube-api-access-9bcnm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-g4wqq\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.434921 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.434821    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-vn5mx\""
Sep 18 00:12:39.443142 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.443121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:39.719871 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.719846    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"]
Sep 18 00:12:39.721219 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:12:39.721195    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d85e89_5652_4427_92a3_2d74f1d28610.slice/crio-5d114b818b383e15666e7ad57d6bbb63b1a378bca94ae55efc4445b7730a8704 WatchSource:0}: Error finding container 5d114b818b383e15666e7ad57d6bbb63b1a378bca94ae55efc4445b7730a8704: Status 404 returned error can't find the container with id 5d114b818b383e15666e7ad57d6bbb63b1a378bca94ae55efc4445b7730a8704
Sep 18 00:12:39.726056 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.726036    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Sep 18 00:12:39.930959 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:39.930930    2562 scope.go:117] "RemoveContainer" containerID="37f1c704ede90c002d6a3f3c62adf20a3b12e18f5b2cfdfaaefee53d1306e3b3"
Sep 18 00:12:40.188808 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:40.188772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" event={"ID":"b1d85e89-5652-4427-92a3-2d74f1d28610","Type":"ContainerStarted","Data":"d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420"}
Sep 18 00:12:40.189206 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:40.188825    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" event={"ID":"b1d85e89-5652-4427-92a3-2d74f1d28610","Type":"ContainerStarted","Data":"5d114b818b383e15666e7ad57d6bbb63b1a378bca94ae55efc4445b7730a8704"}
Sep 18 00:12:42.200660 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:42.200627    2562 generic.go:356] "Generic (PLEG): container finished" podID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerID="d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420" exitCode=0
Sep 18 00:12:42.201148 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:42.200712    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" event={"ID":"b1d85e89-5652-4427-92a3-2d74f1d28610","Type":"ContainerDied","Data":"d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420"}
Sep 18 00:12:43.208382 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:43.208349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" event={"ID":"b1d85e89-5652-4427-92a3-2d74f1d28610","Type":"ContainerStarted","Data":"821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee"}
Sep 18 00:12:43.239327 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:43.234565    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podStartSLOduration=1.9293873430000001 podStartE2EDuration="4.234543565s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.305156222s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:12:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:12:43.229575407 +0000 UTC m=+3724.324102442" watchObservedRunningTime="2026-09-18 00:12:43.234543565 +0000 UTC m=+3724.329070601"
Sep 18 00:12:43.444262 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:43.444230    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:49.444110 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:49.444068    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:12:49.445582 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:49.445550    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:12:59.444157 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:12:59.444111    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:13:09.444016 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:09.443967    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:13:19.444479 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:19.444428    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:13:27.325386 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.325345    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 00:13:27.329141 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.329110    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.331736 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.331714    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Sep 18 00:13:27.331936 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.331914    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-4cwsc\""
Sep 18 00:13:27.332039 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.331935    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 18 00:13:27.332039 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.331982    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Sep 18 00:13:27.333120 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.333102    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-4lzz4\""
Sep 18 00:13:27.333203 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.333140    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 18 00:13:27.333203 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.333149    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 18 00:13:27.340392 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.340365    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 00:13:27.391937 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.391882    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.392184 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.391948    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.392184 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.391986    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.392184 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.392001    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.392184 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.392058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.392184 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.392123    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.392418 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.392200    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.392418 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.392232    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52l9p\" (UniqueName: \"kubernetes.io/projected/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kube-api-access-52l9p\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493087 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493046    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493109    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493129    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52l9p\" (UniqueName: \"kubernetes.io/projected/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kube-api-access-52l9p\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493159    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493192    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493267    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493633 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493302    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493633 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493464    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493633 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493542    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.493963 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493939    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.494195 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.493699    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.503377 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.494458    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.503377 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.502117    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.503377 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.502286    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52l9p\" (UniqueName: \"kubernetes.io/projected/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kube-api-access-52l9p\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.503964 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.503936    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.644182 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.644128    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:13:27.921846 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:27.921816    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 00:13:27.923421 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:13:27.923388    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7110de1_e3c1_4332_9c6b_fbe1f7b17659.slice/crio-c78eee49f708bb9d552bd01bcd1dadb1d84922fc2981636ec79038d829eaf80f WatchSource:0}: Error finding container c78eee49f708bb9d552bd01bcd1dadb1d84922fc2981636ec79038d829eaf80f: Status 404 returned error can't find the container with id c78eee49f708bb9d552bd01bcd1dadb1d84922fc2981636ec79038d829eaf80f
Sep 18 00:13:28.454798 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:28.454755    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a7110de1-e3c1-4332-9c6b-fbe1f7b17659","Type":"ContainerStarted","Data":"416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25"}
Sep 18 00:13:28.455326 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:28.454826    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a7110de1-e3c1-4332-9c6b-fbe1f7b17659","Type":"ContainerStarted","Data":"c78eee49f708bb9d552bd01bcd1dadb1d84922fc2981636ec79038d829eaf80f"}
Sep 18 00:13:29.444659 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:29.444608    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:13:30.470539 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:30.470511    2562 generic.go:356] "Generic (PLEG): container finished" podID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerID="416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25" exitCode=0
Sep 18 00:13:30.470916 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:30.470588    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a7110de1-e3c1-4332-9c6b-fbe1f7b17659","Type":"ContainerDied","Data":"416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25"}
Sep 18 00:13:31.479810 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:31.479769    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a7110de1-e3c1-4332-9c6b-fbe1f7b17659","Type":"ContainerStarted","Data":"9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e"}
Sep 18 00:13:31.501286 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:31.501220    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.04263563 podStartE2EDuration="4.501198544s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.458562914s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:13:31.498550759 +0000 UTC m=+3772.593077796" watchObservedRunningTime="2026-09-18 00:13:31.501198544 +0000 UTC m=+3772.595725580"
Sep 18 00:13:39.444475 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:39.444424    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:13:49.444036 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:49.443986    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:13:59.443966 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:13:59.443920    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:14:09.444573 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:09.444536    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:14:19.443861 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:19.443811    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:14:29.444575 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:29.444525    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:14:39.444287 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:39.444219    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:14:49.454722 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:49.454690    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:14:49.463003 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:49.462977    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:14:50.694490 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.694451    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"]
Sep 18 00:14:50.719200 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.719163    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"]
Sep 18 00:14:50.724012 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.723991    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.735057 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.735020    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"]
Sep 18 00:14:50.871139 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871093    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tls-certs\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.871139 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871144    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-model-cache\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.871386 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871213    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-home\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.871386 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871258    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2wt\" (UniqueName: \"kubernetes.io/projected/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kube-api-access-jg2wt\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.871386 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871294    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.871386 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871323    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.871386 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871370    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-dshm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.871577 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.871401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tmp-dir\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972535 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972437    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972535 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972535 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972528    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-dshm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972824 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972555    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tmp-dir\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972824 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972584    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tls-certs\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972824 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972621    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-model-cache\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972824 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972728    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-home\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.972824 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.972762    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2wt\" (UniqueName: \"kubernetes.io/projected/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kube-api-access-jg2wt\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.973185 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.973135    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tmp-dir\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.973322 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.973218    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-home\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.973397 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.973379    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-model-cache\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.973457 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.973401    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-cabundle-cert\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.973553 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.973527    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kserve-provision-location\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.976258 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.976094    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-dshm\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.976620 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.976594    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tls-certs\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:50.981683 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:50.981663    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2wt\" (UniqueName: \"kubernetes.io/projected/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kube-api-access-jg2wt\") pod \"prestop-hook-test-kserve-5f8cb4d55b-w5x6j\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:51.036765 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:51.036721    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:14:51.318726 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:51.318688    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"]
Sep 18 00:14:51.320219 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:14:51.320186    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfa0ce5_b1ba_43da_a522_0ac60eb8b74d.slice/crio-01b9a47aa3eb740dd21de45cd2d2a390bb9a5996aa5a53f1a9e38b8e81c10d13 WatchSource:0}: Error finding container 01b9a47aa3eb740dd21de45cd2d2a390bb9a5996aa5a53f1a9e38b8e81c10d13: Status 404 returned error can't find the container with id 01b9a47aa3eb740dd21de45cd2d2a390bb9a5996aa5a53f1a9e38b8e81c10d13
Sep 18 00:14:51.962972 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:51.962929    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" event={"ID":"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d","Type":"ContainerStarted","Data":"f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5"}
Sep 18 00:14:51.963354 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:51.962999    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" event={"ID":"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d","Type":"ContainerStarted","Data":"01b9a47aa3eb740dd21de45cd2d2a390bb9a5996aa5a53f1a9e38b8e81c10d13"}
Sep 18 00:14:53.977124 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:53.977090    2562 generic.go:356] "Generic (PLEG): container finished" podID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerID="f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5" exitCode=0
Sep 18 00:14:53.977488 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:53.977163    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" event={"ID":"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d","Type":"ContainerDied","Data":"f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5"}
Sep 18 00:14:54.985153 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:54.985108    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" event={"ID":"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d","Type":"ContainerStarted","Data":"1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4"}
Sep 18 00:14:55.010158 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:55.010062    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" podStartSLOduration=2.64847447 podStartE2EDuration="5.010031501s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.361557031s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:14:55.006324171 +0000 UTC m=+3856.100851207" watchObservedRunningTime="2026-09-18 00:14:55.010031501 +0000 UTC m=+3856.104558538"
Sep 18 00:14:55.037637 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:14:55.037592    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:15:01.037704 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:01.037665    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:15:01.039645 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:01.039594    2562 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Sep 18 00:15:05.978878 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:05.978827    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" containerID="cri-o://821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee" gracePeriod=45
Sep 18 00:15:06.455948 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:06.455874    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Sep 18 00:15:07.034266 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.034238    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:15:07.060131 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.060030    2562 generic.go:356] "Generic (PLEG): container finished" podID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerID="821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee" exitCode=0
Sep 18 00:15:07.060273 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.060131    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" event={"ID":"b1d85e89-5652-4427-92a3-2d74f1d28610","Type":"ContainerDied","Data":"821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee"}
Sep 18 00:15:07.060273 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.060179    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq" event={"ID":"b1d85e89-5652-4427-92a3-2d74f1d28610","Type":"ContainerDied","Data":"5d114b818b383e15666e7ad57d6bbb63b1a378bca94ae55efc4445b7730a8704"}
Sep 18 00:15:07.060273 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.060212    2562 scope.go:117] "RemoveContainer" containerID="821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee"
Sep 18 00:15:07.060396 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.060380    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"
Sep 18 00:15:07.072782 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.072717    2562 scope.go:117] "RemoveContainer" containerID="d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420"
Sep 18 00:15:07.083119 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.083097    2562 scope.go:117] "RemoveContainer" containerID="821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee"
Sep 18 00:15:07.083456 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:15:07.083435    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee\": container with ID starting with 821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee not found: ID does not exist" containerID="821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee"
Sep 18 00:15:07.083534 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.083465    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee"} err="failed to get container status \"821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee\": rpc error: code = NotFound desc = could not find container \"821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee\": container with ID starting with 821e6bb3d0d5cc85a58efd3b3cdaa29db0a5c148c188854e7682655285dbf3ee not found: ID does not exist"
Sep 18 00:15:07.083534 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.083485    2562 scope.go:117] "RemoveContainer" containerID="d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420"
Sep 18 00:15:07.083810 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:15:07.083791    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420\": container with ID starting with d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420 not found: ID does not exist" containerID="d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420"
Sep 18 00:15:07.083810 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.083810    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420"} err="failed to get container status \"d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420\": rpc error: code = NotFound desc = could not find container \"d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420\": container with ID starting with d69a4f88acef8405c5258ef6ea711fc9478c55f459b1ae3ea181965f0a2f5420 not found: ID does not exist"
Sep 18 00:15:07.228365 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228328    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-dshm\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.228568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228413    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-kserve-provision-location\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.228568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228449    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-tmp-dir\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.228568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228481    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d85e89-5652-4427-92a3-2d74f1d28610-tls-certs\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.228568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228507    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bcnm\" (UniqueName: \"kubernetes.io/projected/b1d85e89-5652-4427-92a3-2d74f1d28610-kube-api-access-9bcnm\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.228568 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228536    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-model-cache\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.228857 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228572    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-home\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.228857 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228638    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d85e89-5652-4427-92a3-2d74f1d28610-cabundle-cert\") pod \"b1d85e89-5652-4427-92a3-2d74f1d28610\" (UID: \"b1d85e89-5652-4427-92a3-2d74f1d28610\") "
Sep 18 00:15:07.229075 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.228996    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:07.229285 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.229242    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d85e89-5652-4427-92a3-2d74f1d28610-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:15:07.229980 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.229934    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-home" (OuterVolumeSpecName: "home") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:07.231370 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.231337    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-dshm" (OuterVolumeSpecName: "dshm") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:07.231753 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.231672    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1d85e89-5652-4427-92a3-2d74f1d28610-kube-api-access-9bcnm" (OuterVolumeSpecName: "kube-api-access-9bcnm") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "kube-api-access-9bcnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:15:07.231753 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.231722    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1d85e89-5652-4427-92a3-2d74f1d28610-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:15:07.244058 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.244011    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:07.253158 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.253124    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1d85e89-5652-4427-92a3-2d74f1d28610" (UID: "b1d85e89-5652-4427-92a3-2d74f1d28610"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330153    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330185    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330194    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1d85e89-5652-4427-92a3-2d74f1d28610-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330203    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330212    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330225    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d85e89-5652-4427-92a3-2d74f1d28610-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330239    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1d85e89-5652-4427-92a3-2d74f1d28610-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.330265 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.330246    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bcnm\" (UniqueName: \"kubernetes.io/projected/b1d85e89-5652-4427-92a3-2d74f1d28610-kube-api-access-9bcnm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:07.384654 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.384600    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"]
Sep 18 00:15:07.388691 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.388656    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-g4wqq"]
Sep 18 00:15:07.419893 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:07.419851    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" path="/var/lib/kubelet/pods/b1d85e89-5652-4427-92a3-2d74f1d28610/volumes"
Sep 18 00:15:10.437294 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:10.437253    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"]
Sep 18 00:15:25.475372 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:25.475317    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerName="main" containerID="cri-o://1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4" gracePeriod=45
Sep 18 00:15:32.868507 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:32.868468    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 00:15:47.891166 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:47.891120    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerName="main" containerID="cri-o://9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e" gracePeriod=45
Sep 18 00:15:48.630258 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.630232    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:15:48.696446 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696363    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-cabundle-cert\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.696446 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696415    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-model-cache\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.696673 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696451    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kserve-provision-location\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.696673 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696491    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52l9p\" (UniqueName: \"kubernetes.io/projected/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kube-api-access-52l9p\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.696673 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696524    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-dshm\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.696673 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696548    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tls-certs\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.696673 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696574    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tmp-dir\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.696673 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696605    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-home\") pod \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\" (UID: \"a7110de1-e3c1-4332-9c6b-fbe1f7b17659\") "
Sep 18 00:15:48.697032 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696830    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:48.697032 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.696853    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:15:48.697140 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.697006    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:48.697140 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.697057    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:48.697296 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.697272    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-home" (OuterVolumeSpecName: "home") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:48.699551 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.699512    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kube-api-access-52l9p" (OuterVolumeSpecName: "kube-api-access-52l9p") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "kube-api-access-52l9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:15:48.699777 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.699747    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-dshm" (OuterVolumeSpecName: "dshm") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:48.699777 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.699752    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:15:48.707603 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.707576    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:48.717527 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.717491    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7110de1-e3c1-4332-9c6b-fbe1f7b17659" (UID: "a7110de1-e3c1-4332-9c6b-fbe1f7b17659"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:15:48.798283 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.798232    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:48.798283 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.798277    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52l9p\" (UniqueName: \"kubernetes.io/projected/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-kube-api-access-52l9p\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:48.798283 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.798290    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:48.798283 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.798302    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:48.798562 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.798312    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:48.798562 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:48.798320    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7110de1-e3c1-4332-9c6b-fbe1f7b17659-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:15:49.296854 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.296816    2562 generic.go:356] "Generic (PLEG): container finished" podID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerID="9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e" exitCode=0
Sep 18 00:15:49.297269 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.296864    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Sep 18 00:15:49.297269 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.296888    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a7110de1-e3c1-4332-9c6b-fbe1f7b17659","Type":"ContainerDied","Data":"9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e"}
Sep 18 00:15:49.297269 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.296958    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"a7110de1-e3c1-4332-9c6b-fbe1f7b17659","Type":"ContainerDied","Data":"c78eee49f708bb9d552bd01bcd1dadb1d84922fc2981636ec79038d829eaf80f"}
Sep 18 00:15:49.297269 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.296980    2562 scope.go:117] "RemoveContainer" containerID="9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e"
Sep 18 00:15:49.309730 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.309702    2562 scope.go:117] "RemoveContainer" containerID="416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25"
Sep 18 00:15:49.320131 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.320098    2562 scope.go:117] "RemoveContainer" containerID="9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e"
Sep 18 00:15:49.320532 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:15:49.320508    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e\": container with ID starting with 9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e not found: ID does not exist" containerID="9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e"
Sep 18 00:15:49.320616 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.320542    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e"} err="failed to get container status \"9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e\": rpc error: code = NotFound desc = could not find container \"9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e\": container with ID starting with 9f3c3f5a8dd015641b2201a2300066a8fd51ada8e4841da0dabd35f0a0f3b13e not found: ID does not exist"
Sep 18 00:15:49.320616 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.320561    2562 scope.go:117] "RemoveContainer" containerID="416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25"
Sep 18 00:15:49.320864 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:15:49.320846    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25\": container with ID starting with 416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25 not found: ID does not exist" containerID="416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25"
Sep 18 00:15:49.320945 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.320871    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25"} err="failed to get container status \"416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25\": rpc error: code = NotFound desc = could not find container \"416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25\": container with ID starting with 416fe239eed95466591d0b1a9d912a6f9a297530dc624868c1dfe19d642fee25 not found: ID does not exist"
Sep 18 00:15:49.321561 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.321539    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 00:15:49.325571 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.325541    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Sep 18 00:15:49.419031 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:15:49.418998    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" path="/var/lib/kubelet/pods/a7110de1-e3c1-4332-9c6b-fbe1f7b17659/volumes"
Sep 18 00:16:11.126116 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.126089    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5f8cb4d55b-w5x6j_cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d/main/0.log"
Sep 18 00:16:11.128849 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.128828    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:16:11.195012 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.194972    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-model-cache\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195035    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tls-certs\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195060    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-home\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195080    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-dshm\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195108    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tmp-dir\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195155    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg2wt\" (UniqueName: \"kubernetes.io/projected/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kube-api-access-jg2wt\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195183    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-cabundle-cert\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195554 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195249    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kserve-provision-location\") pod \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\" (UID: \"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d\") "
Sep 18 00:16:11.195554 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195293    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-model-cache" (OuterVolumeSpecName: "model-cache") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:11.195554 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195516    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.195723 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195585    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-home" (OuterVolumeSpecName: "home") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:11.195723 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195651    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 00:16:11.195861 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.195838    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:11.197412 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.197388    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:16:11.197817 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.197797    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-dshm" (OuterVolumeSpecName: "dshm") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:11.197887 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.197811    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kube-api-access-jg2wt" (OuterVolumeSpecName: "kube-api-access-jg2wt") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "kube-api-access-jg2wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:16:11.214891 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.214853    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" (UID: "cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:16:11.296706 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.296623    2562 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kserve-provision-location\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.296706 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.296659    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.296706 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.296670    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.296706 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.296679    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.296706 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.296686    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.296706 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.296695    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg2wt\" (UniqueName: \"kubernetes.io/projected/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-kube-api-access-jg2wt\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.296706 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.296703    2562 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d-cabundle-cert\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:16:11.419585 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.419557    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-5f8cb4d55b-w5x6j_cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d/main/0.log"
Sep 18 00:16:11.419989 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.419968    2562 generic.go:356] "Generic (PLEG): container finished" podID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerID="1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4" exitCode=137
Sep 18 00:16:11.420065 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.420024    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"
Sep 18 00:16:11.420065 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.420037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" event={"ID":"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d","Type":"ContainerDied","Data":"1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4"}
Sep 18 00:16:11.420162 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.420075    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j" event={"ID":"cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d","Type":"ContainerDied","Data":"01b9a47aa3eb740dd21de45cd2d2a390bb9a5996aa5a53f1a9e38b8e81c10d13"}
Sep 18 00:16:11.420162 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.420097    2562 scope.go:117] "RemoveContainer" containerID="1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4"
Sep 18 00:16:11.430466 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.430446    2562 scope.go:117] "RemoveContainer" containerID="f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5"
Sep 18 00:16:11.438346 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.438326    2562 scope.go:117] "RemoveContainer" containerID="1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4"
Sep 18 00:16:11.438593 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:16:11.438573    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4\": container with ID starting with 1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4 not found: ID does not exist" containerID="1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4"
Sep 18 00:16:11.438641 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.438601    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4"} err="failed to get container status \"1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4\": rpc error: code = NotFound desc = could not find container \"1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4\": container with ID starting with 1565e2b989f77d34dd66b28b2837c8031578949789ced54752c42c1a661907a4 not found: ID does not exist"
Sep 18 00:16:11.438641 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.438618    2562 scope.go:117] "RemoveContainer" containerID="f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5"
Sep 18 00:16:11.438857 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:16:11.438840    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5\": container with ID starting with f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5 not found: ID does not exist" containerID="f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5"
Sep 18 00:16:11.438917 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.438861    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5"} err="failed to get container status \"f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5\": rpc error: code = NotFound desc = could not find container \"f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5\": container with ID starting with f4dd1b2698dd09318295887735e491128a5da4a26c2ff410b4e7c272613359e5 not found: ID does not exist"
Sep 18 00:16:11.444666 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.444640    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"]
Sep 18 00:16:11.451450 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:11.451413    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-5f8cb4d55b-w5x6j"]
Sep 18 00:16:13.419009 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:13.418975    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" path="/var/lib/kubelet/pods/cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d/volumes"
Sep 18 00:16:38.293861 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.293775    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"]
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294156    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="storage-initializer"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294168    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="storage-initializer"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294178    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerName="main"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294184    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerName="main"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294190    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerName="storage-initializer"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294196    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerName="storage-initializer"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294207    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerName="main"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294212    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerName="main"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294220    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerName="storage-initializer"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294225    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerName="storage-initializer"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294231    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main"
Sep 18 00:16:38.294239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294235    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main"
Sep 18 00:16:38.294613 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294287    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d85e89-5652-4427-92a3-2d74f1d28610" containerName="main"
Sep 18 00:16:38.294613 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294295    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7110de1-e3c1-4332-9c6b-fbe1f7b17659" containerName="main"
Sep 18 00:16:38.294613 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.294303    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbfa0ce5-b1ba-43da-a522-0ac60eb8b74d" containerName="main"
Sep 18 00:16:38.297199 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.297180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.300051 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.300026    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 18 00:16:38.300218 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.300067    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 18 00:16:38.301208 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.301187    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 18 00:16:38.301521 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.301502    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-lj9wl\""
Sep 18 00:16:38.306565 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.306540    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"]
Sep 18 00:16:38.436491 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.436456    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dcfaf9f-328e-458c-b085-34e4702b6acd-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.436658 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.436505    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcnjd\" (UniqueName: \"kubernetes.io/projected/2dcfaf9f-328e-458c-b085-34e4702b6acd-kube-api-access-rcnjd\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.436658 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.436545    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.436658 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.436589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.436658 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.436614    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.436822 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.436655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.537790 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.537752    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538016 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.537806    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538016 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.537833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538016 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.537862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538016 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.537957    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dcfaf9f-328e-458c-b085-34e4702b6acd-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538016 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.538013    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcnjd\" (UniqueName: \"kubernetes.io/projected/2dcfaf9f-328e-458c-b085-34e4702b6acd-kube-api-access-rcnjd\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538319 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.538294    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538367 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.538295    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.538509 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.538482    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.540323 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.540307    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.540611 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.540592    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dcfaf9f-328e-458c-b085-34e4702b6acd-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.546266 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.546211    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcnjd\" (UniqueName: \"kubernetes.io/projected/2dcfaf9f-328e-458c-b085-34e4702b6acd-kube-api-access-rcnjd\") pod \"rolling-upgrade-test-kserve-74b79d7f77-7tswb\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.608464 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.608425    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:38.884925 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:38.884840    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"]
Sep 18 00:16:38.890950 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:16:38.890881    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcfaf9f_328e_458c_b085_34e4702b6acd.slice/crio-95a5820ce9a0d5391ec4ac8985e7e358b0a004c5fc68721fe7d08eb958227352 WatchSource:0}: Error finding container 95a5820ce9a0d5391ec4ac8985e7e358b0a004c5fc68721fe7d08eb958227352: Status 404 returned error can't find the container with id 95a5820ce9a0d5391ec4ac8985e7e358b0a004c5fc68721fe7d08eb958227352
Sep 18 00:16:39.570517 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:39.570473    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" event={"ID":"2dcfaf9f-328e-458c-b085-34e4702b6acd","Type":"ContainerStarted","Data":"120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73"}
Sep 18 00:16:39.570916 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:39.570533    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" event={"ID":"2dcfaf9f-328e-458c-b085-34e4702b6acd","Type":"ContainerStarted","Data":"95a5820ce9a0d5391ec4ac8985e7e358b0a004c5fc68721fe7d08eb958227352"}
Sep 18 00:16:39.592259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:39.592196    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" podStartSLOduration=1.592181054 podStartE2EDuration="1.592181054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:16:39.589670282 +0000 UTC m=+3960.684197319" watchObservedRunningTime="2026-09-18 00:16:39.592181054 +0000 UTC m=+3960.686708114"
Sep 18 00:16:39.609434 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:39.609395    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:48.608955 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:48.608889    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:48.622269 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:48.622240    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:16:48.633418 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:16:48.633395    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:17:23.419583 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.419541    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"]
Sep 18 00:17:23.428819 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:17:23.428778    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 00:17:23.428819 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:17:23.428819 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 00:17:23.428819 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73" cmd=["/bin/sleep","15"]
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.428831    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" message="" err=<
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:17:23.428843    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" podUID="2dcfaf9f-328e-458c-b085-34e4702b6acd" containerName="main" containerID="cri-o://120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73"
Sep 18 00:17:23.429126 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.428879    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" podUID="2dcfaf9f-328e-458c-b085-34e4702b6acd" containerName="main" containerID="cri-o://120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73" gracePeriod=60
Sep 18 00:17:23.623685 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.623642    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" podUID="2dcfaf9f-328e-458c-b085-34e4702b6acd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Sep 18 00:17:23.772999 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.772974    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:17:23.790414 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.790306    2562 generic.go:356] "Generic (PLEG): container finished" podID="2dcfaf9f-328e-458c-b085-34e4702b6acd" containerID="120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73" exitCode=0
Sep 18 00:17:23.790414 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.790354    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" event={"ID":"2dcfaf9f-328e-458c-b085-34e4702b6acd","Type":"ContainerDied","Data":"120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73"}
Sep 18 00:17:23.790414 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.790372    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"
Sep 18 00:17:23.790414 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.790410    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb" event={"ID":"2dcfaf9f-328e-458c-b085-34e4702b6acd","Type":"ContainerDied","Data":"95a5820ce9a0d5391ec4ac8985e7e358b0a004c5fc68721fe7d08eb958227352"}
Sep 18 00:17:23.790774 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.790441    2562 scope.go:117] "RemoveContainer" containerID="120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73"
Sep 18 00:17:23.801772 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.801744    2562 scope.go:117] "RemoveContainer" containerID="120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73"
Sep 18 00:17:23.802149 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:17:23.802126    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73\": container with ID starting with 120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73 not found: ID does not exist" containerID="120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73"
Sep 18 00:17:23.802226 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.802156    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73"} err="failed to get container status \"120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73\": rpc error: code = NotFound desc = could not find container \"120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73\": container with ID starting with 120e73d189f131cc5c72e841ee05ff549f71861e5dd3b49e997c1e37cd04ae73 not found: ID does not exist"
Sep 18 00:17:23.918256 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918219    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dcfaf9f-328e-458c-b085-34e4702b6acd-tls-certs\") pod \"2dcfaf9f-328e-458c-b085-34e4702b6acd\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") "
Sep 18 00:17:23.918256 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918262    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-model-cache\") pod \"2dcfaf9f-328e-458c-b085-34e4702b6acd\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") "
Sep 18 00:17:23.918530 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918298    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-home\") pod \"2dcfaf9f-328e-458c-b085-34e4702b6acd\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") "
Sep 18 00:17:23.918530 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918336    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcnjd\" (UniqueName: \"kubernetes.io/projected/2dcfaf9f-328e-458c-b085-34e4702b6acd-kube-api-access-rcnjd\") pod \"2dcfaf9f-328e-458c-b085-34e4702b6acd\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") "
Sep 18 00:17:23.918530 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918373    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-tmp-dir\") pod \"2dcfaf9f-328e-458c-b085-34e4702b6acd\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") "
Sep 18 00:17:23.918530 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918418    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-dshm\") pod \"2dcfaf9f-328e-458c-b085-34e4702b6acd\" (UID: \"2dcfaf9f-328e-458c-b085-34e4702b6acd\") "
Sep 18 00:17:23.918747 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918563    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-model-cache" (OuterVolumeSpecName: "model-cache") pod "2dcfaf9f-328e-458c-b085-34e4702b6acd" (UID: "2dcfaf9f-328e-458c-b085-34e4702b6acd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:23.918747 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918607    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-home" (OuterVolumeSpecName: "home") pod "2dcfaf9f-328e-458c-b085-34e4702b6acd" (UID: "2dcfaf9f-328e-458c-b085-34e4702b6acd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:23.918747 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918691    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2dcfaf9f-328e-458c-b085-34e4702b6acd" (UID: "2dcfaf9f-328e-458c-b085-34e4702b6acd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:23.918747 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918709    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:17:23.918747 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.918724    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:17:23.920718 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.920685    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-dshm" (OuterVolumeSpecName: "dshm") pod "2dcfaf9f-328e-458c-b085-34e4702b6acd" (UID: "2dcfaf9f-328e-458c-b085-34e4702b6acd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:17:23.920825 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.920714    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcfaf9f-328e-458c-b085-34e4702b6acd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2dcfaf9f-328e-458c-b085-34e4702b6acd" (UID: "2dcfaf9f-328e-458c-b085-34e4702b6acd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:17:23.920825 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:23.920746    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcfaf9f-328e-458c-b085-34e4702b6acd-kube-api-access-rcnjd" (OuterVolumeSpecName: "kube-api-access-rcnjd") pod "2dcfaf9f-328e-458c-b085-34e4702b6acd" (UID: "2dcfaf9f-328e-458c-b085-34e4702b6acd"). InnerVolumeSpecName "kube-api-access-rcnjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:17:24.019399 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:24.019296    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dcfaf9f-328e-458c-b085-34e4702b6acd-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:17:24.019399 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:24.019333    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcnjd\" (UniqueName: \"kubernetes.io/projected/2dcfaf9f-328e-458c-b085-34e4702b6acd-kube-api-access-rcnjd\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:17:24.019399 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:24.019347    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:17:24.019399 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:24.019361    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dcfaf9f-328e-458c-b085-34e4702b6acd-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:17:24.112634 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:24.112599    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"]
Sep 18 00:17:24.114529 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:24.114505    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-7tswb"]
Sep 18 00:17:25.418505 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:17:25.418470    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dcfaf9f-328e-458c-b085-34e4702b6acd" path="/var/lib/kubelet/pods/2dcfaf9f-328e-458c-b085-34e4702b6acd/volumes"
Sep 18 00:18:28.582445 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.582409    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"]
Sep 18 00:18:28.582853 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.582752    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dcfaf9f-328e-458c-b085-34e4702b6acd" containerName="main"
Sep 18 00:18:28.582853 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.582764    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dcfaf9f-328e-458c-b085-34e4702b6acd" containerName="main"
Sep 18 00:18:28.582853 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.582835    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dcfaf9f-328e-458c-b085-34e4702b6acd" containerName="main"
Sep 18 00:18:28.585449 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.585430    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.588132 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.588108    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 18 00:18:28.589229 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.589207    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 18 00:18:28.589229 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.589223    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-tr5gd\""
Sep 18 00:18:28.589445 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.589229    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 18 00:18:28.594523 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.594496    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"]
Sep 18 00:18:28.655024 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.654983    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.655024 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.655030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.655255 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.655064    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5204c99-2e95-45df-8901-cc9aad10ab8d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.655255 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.655110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.655255 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.655161    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.655255 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.655189    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw28c\" (UniqueName: \"kubernetes.io/projected/f5204c99-2e95-45df-8901-cc9aad10ab8d-kube-api-access-nw28c\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.756423 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.756376    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5204c99-2e95-45df-8901-cc9aad10ab8d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.756423 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.756424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.756681 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.756473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.756681 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.756501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw28c\" (UniqueName: \"kubernetes.io/projected/f5204c99-2e95-45df-8901-cc9aad10ab8d-kube-api-access-nw28c\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.756681 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.756549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.756681 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.756576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.757009 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.756985    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.757009 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.757002    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.757142 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.757118    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.758821 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.758800    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.758988 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.758970    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5204c99-2e95-45df-8901-cc9aad10ab8d-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.764239 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.764204    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw28c\" (UniqueName: \"kubernetes.io/projected/f5204c99-2e95-45df-8901-cc9aad10ab8d-kube-api-access-nw28c\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:28.896595 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:28.896479    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:29.165986 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:29.165959    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"]
Sep 18 00:18:29.167639 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:18:29.167610    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5204c99_2e95_45df_8901_cc9aad10ab8d.slice/crio-97cc839fd61da47feeae8da71d383d86ed57cbf5c64caddbb7297257d81a0f86 WatchSource:0}: Error finding container 97cc839fd61da47feeae8da71d383d86ed57cbf5c64caddbb7297257d81a0f86: Status 404 returned error can't find the container with id 97cc839fd61da47feeae8da71d383d86ed57cbf5c64caddbb7297257d81a0f86
Sep 18 00:18:30.126202 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:30.126167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" event={"ID":"f5204c99-2e95-45df-8901-cc9aad10ab8d","Type":"ContainerStarted","Data":"675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b"}
Sep 18 00:18:30.126607 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:30.126216    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" event={"ID":"f5204c99-2e95-45df-8901-cc9aad10ab8d","Type":"ContainerStarted","Data":"97cc839fd61da47feeae8da71d383d86ed57cbf5c64caddbb7297257d81a0f86"}
Sep 18 00:18:30.145651 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:30.145596    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" podStartSLOduration=2.145580892 podStartE2EDuration="2.145580892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:18:30.142635802 +0000 UTC m=+4071.237162829" watchObservedRunningTime="2026-09-18 00:18:30.145580892 +0000 UTC m=+4071.240107927"
Sep 18 00:18:30.897187 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:30.897141    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:38.896762 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:38.896692    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:38.909831 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:38.909797    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:18:39.186688 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:18:39.186599    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:19:02.827696 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:02.827661    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"]
Sep 18 00:19:02.837001 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:19:02.836961    2562 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 18 00:19:02.837001 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:19:02.837001 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 00:19:02.837001 ip-10-0-132-155 kubenswrapper[2562]:  > containerID="675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b" cmd=["/bin/sleep","15"]
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:02.837022    2562 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" message="" err=<
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]:  >
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:19:02.837039    2562 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]:         , stdout: , stderr: , exit code -1
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" podUID="f5204c99-2e95-45df-8901-cc9aad10ab8d" containerName="main" containerID="cri-o://675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b"
Sep 18 00:19:02.837329 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:02.837095    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" podUID="f5204c99-2e95-45df-8901-cc9aad10ab8d" containerName="main" containerID="cri-o://675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b" gracePeriod=60
Sep 18 00:19:03.192183 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.192152    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:19:03.258835 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.258794    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nw28c\" (UniqueName: \"kubernetes.io/projected/f5204c99-2e95-45df-8901-cc9aad10ab8d-kube-api-access-nw28c\") pod \"f5204c99-2e95-45df-8901-cc9aad10ab8d\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") "
Sep 18 00:19:03.259064 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.258853    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-home\") pod \"f5204c99-2e95-45df-8901-cc9aad10ab8d\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") "
Sep 18 00:19:03.259064 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.258881    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-model-cache\") pod \"f5204c99-2e95-45df-8901-cc9aad10ab8d\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") "
Sep 18 00:19:03.259064 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.258927    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5204c99-2e95-45df-8901-cc9aad10ab8d-tls-certs\") pod \"f5204c99-2e95-45df-8901-cc9aad10ab8d\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") "
Sep 18 00:19:03.259064 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.258957    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-dshm\") pod \"f5204c99-2e95-45df-8901-cc9aad10ab8d\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") "
Sep 18 00:19:03.259064 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.258981    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-tmp-dir\") pod \"f5204c99-2e95-45df-8901-cc9aad10ab8d\" (UID: \"f5204c99-2e95-45df-8901-cc9aad10ab8d\") "
Sep 18 00:19:03.259343 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.259219    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5204c99-2e95-45df-8901-cc9aad10ab8d" (UID: "f5204c99-2e95-45df-8901-cc9aad10ab8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:19:03.259343 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.259236    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-home" (OuterVolumeSpecName: "home") pod "f5204c99-2e95-45df-8901-cc9aad10ab8d" (UID: "f5204c99-2e95-45df-8901-cc9aad10ab8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:19:03.259343 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.259302    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5204c99-2e95-45df-8901-cc9aad10ab8d" (UID: "f5204c99-2e95-45df-8901-cc9aad10ab8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:19:03.261372 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.261349    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5204c99-2e95-45df-8901-cc9aad10ab8d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5204c99-2e95-45df-8901-cc9aad10ab8d" (UID: "f5204c99-2e95-45df-8901-cc9aad10ab8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 00:19:03.261372 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.261363    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5204c99-2e95-45df-8901-cc9aad10ab8d-kube-api-access-nw28c" (OuterVolumeSpecName: "kube-api-access-nw28c") pod "f5204c99-2e95-45df-8901-cc9aad10ab8d" (UID: "f5204c99-2e95-45df-8901-cc9aad10ab8d"). InnerVolumeSpecName "kube-api-access-nw28c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:19:03.261508 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.261371    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-dshm" (OuterVolumeSpecName: "dshm") pod "f5204c99-2e95-45df-8901-cc9aad10ab8d" (UID: "f5204c99-2e95-45df-8901-cc9aad10ab8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:19:03.307447 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.307414    2562 generic.go:356] "Generic (PLEG): container finished" podID="f5204c99-2e95-45df-8901-cc9aad10ab8d" containerID="675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b" exitCode=0
Sep 18 00:19:03.307620 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.307462    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"
Sep 18 00:19:03.307620 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.307505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" event={"ID":"f5204c99-2e95-45df-8901-cc9aad10ab8d","Type":"ContainerDied","Data":"675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b"}
Sep 18 00:19:03.307620 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.307562    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" event={"ID":"f5204c99-2e95-45df-8901-cc9aad10ab8d","Type":"ContainerDied","Data":"97cc839fd61da47feeae8da71d383d86ed57cbf5c64caddbb7297257d81a0f86"}
Sep 18 00:19:03.307620 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.307590    2562 scope.go:117] "RemoveContainer" containerID="675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b"
Sep 18 00:19:03.317418 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.317398    2562 scope.go:117] "RemoveContainer" containerID="675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b"
Sep 18 00:19:03.317711 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:19:03.317688    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b\": container with ID starting with 675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b not found: ID does not exist" containerID="675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b"
Sep 18 00:19:03.317762 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.317720    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b"} err="failed to get container status \"675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b\": rpc error: code = NotFound desc = could not find container \"675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b\": container with ID starting with 675933ea264639d91fc0d39dcde296407a180c79fea755a1b09d840eb097583b not found: ID does not exist"
Sep 18 00:19:03.329031 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.328997    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"]
Sep 18 00:19:03.332911 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.332875    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh"]
Sep 18 00:19:03.360285 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.360210    2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-dshm\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:19:03.360285 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.360238    2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-tmp-dir\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:19:03.360285 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.360251    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw28c\" (UniqueName: \"kubernetes.io/projected/f5204c99-2e95-45df-8901-cc9aad10ab8d-kube-api-access-nw28c\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:19:03.360285 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.360261    2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-home\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:19:03.360285 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.360271    2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5204c99-2e95-45df-8901-cc9aad10ab8d-model-cache\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:19:03.360285 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.360278    2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5204c99-2e95-45df-8901-cc9aad10ab8d-tls-certs\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:19:03.418883 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:03.418853    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5204c99-2e95-45df-8901-cc9aad10ab8d" path="/var/lib/kubelet/pods/f5204c99-2e95-45df-8901-cc9aad10ab8d/volumes"
Sep 18 00:19:04.176564 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:04.176507    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-cg2bh" podUID="f5204c99-2e95-45df-8901-cc9aad10ab8d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 18 00:19:55.925219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.925180    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kskgr/must-gather-p6bpx"]
Sep 18 00:19:55.925630 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.925523    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5204c99-2e95-45df-8901-cc9aad10ab8d" containerName="main"
Sep 18 00:19:55.925630 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.925535    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5204c99-2e95-45df-8901-cc9aad10ab8d" containerName="main"
Sep 18 00:19:55.925630 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.925596    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5204c99-2e95-45df-8901-cc9aad10ab8d" containerName="main"
Sep 18 00:19:55.929719 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.929701    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:55.932966 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.932930    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kskgr\"/\"kube-root-ca.crt\""
Sep 18 00:19:55.933119 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.932998    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kskgr\"/\"openshift-service-ca.crt\""
Sep 18 00:19:55.933240 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.933223    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kskgr\"/\"default-dockercfg-2zq6m\""
Sep 18 00:19:55.943738 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:55.943714    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kskgr/must-gather-p6bpx"]
Sep 18 00:19:56.002193 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.002140    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg5qw\" (UniqueName: \"kubernetes.io/projected/227c7369-8f99-4e42-8522-ff0de0052175-kube-api-access-fg5qw\") pod \"must-gather-p6bpx\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") " pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:56.002193 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.002185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/227c7369-8f99-4e42-8522-ff0de0052175-must-gather-output\") pod \"must-gather-p6bpx\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") " pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:56.103240 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.103190    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg5qw\" (UniqueName: \"kubernetes.io/projected/227c7369-8f99-4e42-8522-ff0de0052175-kube-api-access-fg5qw\") pod \"must-gather-p6bpx\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") " pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:56.103240 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.103249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/227c7369-8f99-4e42-8522-ff0de0052175-must-gather-output\") pod \"must-gather-p6bpx\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") " pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:56.103581 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.103565    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/227c7369-8f99-4e42-8522-ff0de0052175-must-gather-output\") pod \"must-gather-p6bpx\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") " pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:56.112306 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.112277    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg5qw\" (UniqueName: \"kubernetes.io/projected/227c7369-8f99-4e42-8522-ff0de0052175-kube-api-access-fg5qw\") pod \"must-gather-p6bpx\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") " pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:56.238439 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.238369    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:19:56.507030 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.506998    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kskgr/must-gather-p6bpx"]
Sep 18 00:19:56.512330 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:19:56.512293    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod227c7369_8f99_4e42_8522_ff0de0052175.slice/crio-155901dcebd92f6965b03279bbdedc0bb1d7f4e5a95637efb832240aca1feb16 WatchSource:0}: Error finding container 155901dcebd92f6965b03279bbdedc0bb1d7f4e5a95637efb832240aca1feb16: Status 404 returned error can't find the container with id 155901dcebd92f6965b03279bbdedc0bb1d7f4e5a95637efb832240aca1feb16
Sep 18 00:19:56.513714 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.513696    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 00:19:56.574076 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:19:56.574043    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kskgr/must-gather-p6bpx" event={"ID":"227c7369-8f99-4e42-8522-ff0de0052175","Type":"ContainerStarted","Data":"155901dcebd92f6965b03279bbdedc0bb1d7f4e5a95637efb832240aca1feb16"}
Sep 18 00:20:03.617029 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:03.616991    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kskgr/must-gather-p6bpx" event={"ID":"227c7369-8f99-4e42-8522-ff0de0052175","Type":"ContainerStarted","Data":"a8ab8faeb5c86a4a01a9aef5adec9c16827c862c188ff5672a0f11a72faa5185"}
Sep 18 00:20:03.617479 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:03.617039    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kskgr/must-gather-p6bpx" event={"ID":"227c7369-8f99-4e42-8522-ff0de0052175","Type":"ContainerStarted","Data":"f7d4f38831448e23ab3dcad1822bea02b934493bc072047eedb4a8c494233857"}
Sep 18 00:20:03.635984 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:03.635929    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kskgr/must-gather-p6bpx" podStartSLOduration=2.446488729 podStartE2EDuration="8.63589463s" totalImagesPullingTime="6.189405901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:19:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:20:03.634261097 +0000 UTC m=+4164.728788137" watchObservedRunningTime="2026-09-18 00:20:03.63589463 +0000 UTC m=+4164.730421666"
Sep 18 00:20:20.833268 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:20.833236    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-84x5c_4efc648c-026b-447b-aac8-30f400a3d6af/kuadrant-console-plugin/0.log"
Sep 18 00:20:20.925018 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:20.924977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-j96gz_cac47e37-e3a9-490b-96d7-c72a721c5671/limitador/0.log"
Sep 18 00:20:25.749297 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:25.749261    2562 generic.go:356] "Generic (PLEG): container finished" podID="227c7369-8f99-4e42-8522-ff0de0052175" containerID="f7d4f38831448e23ab3dcad1822bea02b934493bc072047eedb4a8c494233857" exitCode=0
Sep 18 00:20:25.749718 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:25.749332    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kskgr/must-gather-p6bpx" event={"ID":"227c7369-8f99-4e42-8522-ff0de0052175","Type":"ContainerDied","Data":"f7d4f38831448e23ab3dcad1822bea02b934493bc072047eedb4a8c494233857"}
Sep 18 00:20:25.749718 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:25.749676    2562 scope.go:117] "RemoveContainer" containerID="f7d4f38831448e23ab3dcad1822bea02b934493bc072047eedb4a8c494233857"
Sep 18 00:20:26.132360 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.132328    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kskgr_must-gather-p6bpx_227c7369-8f99-4e42-8522-ff0de0052175/gather/0.log"
Sep 18 00:20:26.642563 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.642526    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8545l/must-gather-56ffk"]
Sep 18 00:20:26.645671 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.645647    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:26.649229 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.649208    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8545l\"/\"default-dockercfg-vqjqc\""
Sep 18 00:20:26.650092 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.650071    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8545l\"/\"kube-root-ca.crt\""
Sep 18 00:20:26.650092 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.650084    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8545l\"/\"openshift-service-ca.crt\""
Sep 18 00:20:26.658548 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.658518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8545l/must-gather-56ffk"]
Sep 18 00:20:26.787783 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.787739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26-must-gather-output\") pod \"must-gather-56ffk\" (UID: \"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26\") " pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:26.788219 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.787813    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcfqx\" (UniqueName: \"kubernetes.io/projected/1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26-kube-api-access-jcfqx\") pod \"must-gather-56ffk\" (UID: \"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26\") " pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:26.888653 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.888605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26-must-gather-output\") pod \"must-gather-56ffk\" (UID: \"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26\") " pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:26.888859 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.888678    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcfqx\" (UniqueName: \"kubernetes.io/projected/1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26-kube-api-access-jcfqx\") pod \"must-gather-56ffk\" (UID: \"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26\") " pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:26.889053 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.889031    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26-must-gather-output\") pod \"must-gather-56ffk\" (UID: \"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26\") " pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:26.896781 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.896729    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcfqx\" (UniqueName: \"kubernetes.io/projected/1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26-kube-api-access-jcfqx\") pod \"must-gather-56ffk\" (UID: \"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26\") " pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:26.955691 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:26.955650    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8545l/must-gather-56ffk"
Sep 18 00:20:27.225494 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:27.225465    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8545l/must-gather-56ffk"]
Sep 18 00:20:27.227408 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:20:27.227369    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d4a34c3_b6ef_4208_9bfb_cd1b267b0a26.slice/crio-8bfbdbdaccde4dda4baa85ce974d273eb41b983667dc26e165c673ff91181f52 WatchSource:0}: Error finding container 8bfbdbdaccde4dda4baa85ce974d273eb41b983667dc26e165c673ff91181f52: Status 404 returned error can't find the container with id 8bfbdbdaccde4dda4baa85ce974d273eb41b983667dc26e165c673ff91181f52
Sep 18 00:20:27.762827 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:27.762786    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8545l/must-gather-56ffk" event={"ID":"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26","Type":"ContainerStarted","Data":"8bfbdbdaccde4dda4baa85ce974d273eb41b983667dc26e165c673ff91181f52"}
Sep 18 00:20:28.771230 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:28.771189    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8545l/must-gather-56ffk" event={"ID":"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26","Type":"ContainerStarted","Data":"dfdbec76b52893c8e3b84f4564ba3ca77c6a1558e3819533c656305b8a74ceea"}
Sep 18 00:20:28.771230 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:28.771235    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8545l/must-gather-56ffk" event={"ID":"1d4a34c3-b6ef-4208-9bfb-cd1b267b0a26","Type":"ContainerStarted","Data":"0be9a650d0521cbd04c497725364f7763767a58399a2f36aba24c0f98f2e2ba9"}
Sep 18 00:20:28.788734 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:28.788664    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8545l/must-gather-56ffk" podStartSLOduration=2.099966689 podStartE2EDuration="2.788642145s" totalImagesPullingTime="688.675456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 00:20:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:20:28.78493148 +0000 UTC m=+4189.879458519" watchObservedRunningTime="2026-09-18 00:20:28.788642145 +0000 UTC m=+4189.883169182"
Sep 18 00:20:29.375862 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:29.375823    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d4lvj_9ae1b512-2e62-465c-8d6f-8e782e948808/global-pull-secret-syncer/0.log"
Sep 18 00:20:29.570012 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:29.569974    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s94pm_db20b0b2-8082-4c9e-b2e9-f136c3b2efec/konnectivity-agent/0.log"
Sep 18 00:20:29.593629 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:29.593592    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-155.ec2.internal_52838b2473a80c0fab97ff455ee0daf5/haproxy/0.log"
Sep 18 00:20:31.503276 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.503238    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-kskgr/must-gather-p6bpx"]
Sep 18 00:20:31.504389 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.504022    2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-kskgr/must-gather-p6bpx" podUID="227c7369-8f99-4e42-8522-ff0de0052175" containerName="copy" containerID="cri-o://a8ab8faeb5c86a4a01a9aef5adec9c16827c862c188ff5672a0f11a72faa5185" gracePeriod=2
Sep 18 00:20:31.507148 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:20:31.507042    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p6bpx\" is forbidden: User \"system:node:ip-10-0-132-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kskgr\": no relationship found between node 'ip-10-0-132-155.ec2.internal' and this object" podUID="227c7369-8f99-4e42-8522-ff0de0052175" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:20:31.507327 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.507307    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-kskgr/must-gather-p6bpx"]
Sep 18 00:20:31.806290 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.806056    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kskgr_must-gather-p6bpx_227c7369-8f99-4e42-8522-ff0de0052175/copy/0.log"
Sep 18 00:20:31.806729 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.806555    2562 generic.go:356] "Generic (PLEG): container finished" podID="227c7369-8f99-4e42-8522-ff0de0052175" containerID="a8ab8faeb5c86a4a01a9aef5adec9c16827c862c188ff5672a0f11a72faa5185" exitCode=143
Sep 18 00:20:31.902516 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.902478    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kskgr_must-gather-p6bpx_227c7369-8f99-4e42-8522-ff0de0052175/copy/0.log"
Sep 18 00:20:31.921923 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.916133    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:20:31.921923 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:20:31.919129    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p6bpx\" is forbidden: User \"system:node:ip-10-0-132-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kskgr\": no relationship found between node 'ip-10-0-132-155.ec2.internal' and this object" podUID="227c7369-8f99-4e42-8522-ff0de0052175" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:20:31.942022 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.941218    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/227c7369-8f99-4e42-8522-ff0de0052175-must-gather-output\") pod \"227c7369-8f99-4e42-8522-ff0de0052175\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") "
Sep 18 00:20:31.942022 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.941335    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg5qw\" (UniqueName: \"kubernetes.io/projected/227c7369-8f99-4e42-8522-ff0de0052175-kube-api-access-fg5qw\") pod \"227c7369-8f99-4e42-8522-ff0de0052175\" (UID: \"227c7369-8f99-4e42-8522-ff0de0052175\") "
Sep 18 00:20:31.947439 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.947370    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227c7369-8f99-4e42-8522-ff0de0052175-kube-api-access-fg5qw" (OuterVolumeSpecName: "kube-api-access-fg5qw") pod "227c7369-8f99-4e42-8522-ff0de0052175" (UID: "227c7369-8f99-4e42-8522-ff0de0052175"). InnerVolumeSpecName "kube-api-access-fg5qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 00:20:31.951976 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:31.950778    2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/227c7369-8f99-4e42-8522-ff0de0052175-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "227c7369-8f99-4e42-8522-ff0de0052175" (UID: "227c7369-8f99-4e42-8522-ff0de0052175"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 00:20:32.042451 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:32.042404    2562 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/227c7369-8f99-4e42-8522-ff0de0052175-must-gather-output\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:20:32.042697 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:32.042678    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg5qw\" (UniqueName: \"kubernetes.io/projected/227c7369-8f99-4e42-8522-ff0de0052175-kube-api-access-fg5qw\") on node \"ip-10-0-132-155.ec2.internal\" DevicePath \"\""
Sep 18 00:20:32.816342 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:32.816255    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-kskgr_must-gather-p6bpx_227c7369-8f99-4e42-8522-ff0de0052175/copy/0.log"
Sep 18 00:20:32.816944 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:32.816831    2562 scope.go:117] "RemoveContainer" containerID="a8ab8faeb5c86a4a01a9aef5adec9c16827c862c188ff5672a0f11a72faa5185"
Sep 18 00:20:32.817055 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:32.817033    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:20:32.820769 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:20:32.820735    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p6bpx\" is forbidden: User \"system:node:ip-10-0-132-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kskgr\": no relationship found between node 'ip-10-0-132-155.ec2.internal' and this object" podUID="227c7369-8f99-4e42-8522-ff0de0052175" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:20:32.834336 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:32.834308    2562 scope.go:117] "RemoveContainer" containerID="f7d4f38831448e23ab3dcad1822bea02b934493bc072047eedb4a8c494233857"
Sep 18 00:20:32.841185 ip-10-0-132-155 kubenswrapper[2562]: E0918 00:20:32.841144    2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-p6bpx\" is forbidden: User \"system:node:ip-10-0-132-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-kskgr\": no relationship found between node 'ip-10-0-132-155.ec2.internal' and this object" podUID="227c7369-8f99-4e42-8522-ff0de0052175" pod="openshift-must-gather-kskgr/must-gather-p6bpx"
Sep 18 00:20:33.423096 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:33.423056    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="227c7369-8f99-4e42-8522-ff0de0052175" path="/var/lib/kubelet/pods/227c7369-8f99-4e42-8522-ff0de0052175/volumes"
Sep 18 00:20:34.596179 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:34.596143    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-84x5c_4efc648c-026b-447b-aac8-30f400a3d6af/kuadrant-console-plugin/0.log"
Sep 18 00:20:34.710288 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:34.710252    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-j96gz_cac47e37-e3a9-490b-96d7-c72a721c5671/limitador/0.log"
Sep 18 00:20:36.136864 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.136821    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51c1b7ad-b451-4bbf-bf86-00920f46fdf0/alertmanager/0.log"
Sep 18 00:20:36.157776 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.157733    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51c1b7ad-b451-4bbf-bf86-00920f46fdf0/config-reloader/0.log"
Sep 18 00:20:36.178679 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.178648    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51c1b7ad-b451-4bbf-bf86-00920f46fdf0/kube-rbac-proxy-web/0.log"
Sep 18 00:20:36.202039 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.202007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51c1b7ad-b451-4bbf-bf86-00920f46fdf0/kube-rbac-proxy/0.log"
Sep 18 00:20:36.222311 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.222284    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51c1b7ad-b451-4bbf-bf86-00920f46fdf0/kube-rbac-proxy-metric/0.log"
Sep 18 00:20:36.245864 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.245813    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51c1b7ad-b451-4bbf-bf86-00920f46fdf0/prom-label-proxy/0.log"
Sep 18 00:20:36.266243 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.266216    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51c1b7ad-b451-4bbf-bf86-00920f46fdf0/init-config-reloader/0.log"
Sep 18 00:20:36.308754 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.308611    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-847b8bd8d7-77mnh_16844c76-db20-404d-b5ed-b7fcac6f49db/cluster-monitoring-operator/0.log"
Sep 18 00:20:36.420061 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.419977    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5984459ff8-94w87_7a182c8b-3b46-447e-b0ea-2ea538079226/monitoring-plugin/0.log"
Sep 18 00:20:36.589795 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.589763    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjjtn_890bf2cd-29bf-4c8f-bcdd-474ec8455fc0/node-exporter/0.log"
Sep 18 00:20:36.611669 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.611639    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjjtn_890bf2cd-29bf-4c8f-bcdd-474ec8455fc0/kube-rbac-proxy/0.log"
Sep 18 00:20:36.631572 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.631548    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mjjtn_890bf2cd-29bf-4c8f-bcdd-474ec8455fc0/init-textfile/0.log"
Sep 18 00:20:36.947372 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:36.947337    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-vhm6n_e155d60b-3aa5-47d1-a744-059cf5a07f73/prometheus-operator-admission-webhook/0.log"
Sep 18 00:20:37.052164 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:37.052116    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d5ccdf6-rwbsq_fc275d57-fcd0-4671-91c8-6f0d91c9ac01/thanos-query/0.log"
Sep 18 00:20:37.081940 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:37.081887    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d5ccdf6-rwbsq_fc275d57-fcd0-4671-91c8-6f0d91c9ac01/kube-rbac-proxy-web/0.log"
Sep 18 00:20:37.111634 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:37.111603    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d5ccdf6-rwbsq_fc275d57-fcd0-4671-91c8-6f0d91c9ac01/kube-rbac-proxy/0.log"
Sep 18 00:20:37.141698 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:37.141666    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d5ccdf6-rwbsq_fc275d57-fcd0-4671-91c8-6f0d91c9ac01/prom-label-proxy/0.log"
Sep 18 00:20:37.164992 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:37.164803    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d5ccdf6-rwbsq_fc275d57-fcd0-4671-91c8-6f0d91c9ac01/kube-rbac-proxy-rules/0.log"
Sep 18 00:20:37.189445 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:37.189418    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-579d5ccdf6-rwbsq_fc275d57-fcd0-4671-91c8-6f0d91c9ac01/kube-rbac-proxy-metrics/0.log"
Sep 18 00:20:39.517617 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:39.517582    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-gmqx7_e955fcd2-2eff-4492-87db-c4f4bb1312a2/download-server/0.log"
Sep 18 00:20:40.039103 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.039070    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-qrjg6_4d41971a-8a8e-4003-8d1e-4aba3d5bc846/volume-data-source-validator/0.log"
Sep 18 00:20:40.156409 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.156363    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"]
Sep 18 00:20:40.157212 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.157181    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227c7369-8f99-4e42-8522-ff0de0052175" containerName="copy"
Sep 18 00:20:40.157369 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.157358    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="227c7369-8f99-4e42-8522-ff0de0052175" containerName="copy"
Sep 18 00:20:40.157464 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.157454    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="227c7369-8f99-4e42-8522-ff0de0052175" containerName="gather"
Sep 18 00:20:40.157541 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.157533    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="227c7369-8f99-4e42-8522-ff0de0052175" containerName="gather"
Sep 18 00:20:40.157705 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.157695    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="227c7369-8f99-4e42-8522-ff0de0052175" containerName="gather"
Sep 18 00:20:40.157791 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.157782    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="227c7369-8f99-4e42-8522-ff0de0052175" containerName="copy"
Sep 18 00:20:40.162894 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.162869    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.169620 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.169580    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"]
Sep 18 00:20:40.223893 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.223847    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-sys\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.224101 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.223950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8v5\" (UniqueName: \"kubernetes.io/projected/f58abbba-f70a-4569-b271-125ac99d0cff-kube-api-access-7x8v5\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.224101 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.224000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-podres\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.224101 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.224048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-proc\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.224101 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.224075    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-lib-modules\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.324934 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.324815    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-sys\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.324934 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.324887    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8v5\" (UniqueName: \"kubernetes.io/projected/f58abbba-f70a-4569-b271-125ac99d0cff-kube-api-access-7x8v5\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.325172 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.324965    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-sys\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.325172 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.324975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-podres\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.325172 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.325057    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-proc\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.325172 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.325087    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-lib-modules\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.325400 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.325119    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-podres\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.325400 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.325182    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-proc\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.325400 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.325276    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f58abbba-f70a-4569-b271-125ac99d0cff-lib-modules\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.333102 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.333066    2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8v5\" (UniqueName: \"kubernetes.io/projected/f58abbba-f70a-4569-b271-125ac99d0cff-kube-api-access-7x8v5\") pod \"perf-node-gather-daemonset-8v5wf\" (UID: \"f58abbba-f70a-4569-b271-125ac99d0cff\") " pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.476921 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.476859    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:40.771152 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.771124    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"]
Sep 18 00:20:40.773124 ip-10-0-132-155 kubenswrapper[2562]: W0918 00:20:40.773095    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf58abbba_f70a_4569_b271_125ac99d0cff.slice/crio-e82811237467aba2c7aafc8a9464fd3bb407695c5a5486bd061f5231f6982bcd WatchSource:0}: Error finding container e82811237467aba2c7aafc8a9464fd3bb407695c5a5486bd061f5231f6982bcd: Status 404 returned error can't find the container with id e82811237467aba2c7aafc8a9464fd3bb407695c5a5486bd061f5231f6982bcd
Sep 18 00:20:40.881615 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:40.881581    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf" event={"ID":"f58abbba-f70a-4569-b271-125ac99d0cff","Type":"ContainerStarted","Data":"e82811237467aba2c7aafc8a9464fd3bb407695c5a5486bd061f5231f6982bcd"}
Sep 18 00:20:41.054724 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:41.054616    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nc7w8_fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5/dns/0.log"
Sep 18 00:20:41.072363 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:41.072329    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nc7w8_fbe828f4-79f3-4e4a-b2a7-eaff07c8fee5/kube-rbac-proxy/0.log"
Sep 18 00:20:41.131961 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:41.131932    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9ngsd_3963701d-1a30-4685-99f4-d9a1bc8af35b/dns-node-resolver/0.log"
Sep 18 00:20:41.745038 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:41.745007    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-szjtn_021443d4-2c25-4789-9f74-acc54f1e1095/node-ca/0.log"
Sep 18 00:20:41.890346 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:41.890313    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf" event={"ID":"f58abbba-f70a-4569-b271-125ac99d0cff","Type":"ContainerStarted","Data":"ba1338e8283e21ddc952693695b333055126285c98fa830573a9fa248207cd53"}
Sep 18 00:20:41.890777 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:41.890461    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:41.907254 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:41.907199    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf" podStartSLOduration=1.907182822 podStartE2EDuration="1.907182822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 00:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 00:20:41.904640212 +0000 UTC m=+4202.999167250" watchObservedRunningTime="2026-09-18 00:20:41.907182822 +0000 UTC m=+4203.001709900"
Sep 18 00:20:43.364837 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:43.364810    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-m8nq7_613efdc2-51ea-45ce-b410-861640d21bbf/serve-healthcheck-canary/0.log"
Sep 18 00:20:43.839624 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:43.839591    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-tvnc7_f41801af-ae26-4324-8400-1ace17f3414d/insights-operator/1.log"
Sep 18 00:20:43.840097 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:43.840072    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-tvnc7_f41801af-ae26-4324-8400-1ace17f3414d/insights-operator/0.log"
Sep 18 00:20:43.973719 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:43.973686    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9crng_27628199-2db6-4d59-993b-b126afbe3304/kube-rbac-proxy/0.log"
Sep 18 00:20:43.991020 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:43.990986    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9crng_27628199-2db6-4d59-993b-b126afbe3304/exporter/0.log"
Sep 18 00:20:44.008302 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:44.008267    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9crng_27628199-2db6-4d59-993b-b126afbe3304/extractor/0.log"
Sep 18 00:20:46.855347 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:46.855314    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6944fcc464-kk97f_95538e94-8be7-4862-8873-75294372c415/manager/0.log"
Sep 18 00:20:47.908468 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:47.908441    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8545l/perf-node-gather-daemonset-8v5wf"
Sep 18 00:20:48.458835 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:48.458795    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-d9zkc_d2c26a08-3d59-4e42-8254-83511fc3af92/s3-init/0.log"
Sep 18 00:20:48.477498 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:48.477473    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-d9zkc_d2c26a08-3d59-4e42-8254-83511fc3af92/download-hf-model/0.log"
Sep 18 00:20:53.471019 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:53.470981    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-zqm4k_ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7/migrator/0.log"
Sep 18 00:20:53.488390 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:53.488356    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-zqm4k_ca9b041d-da68-44ef-9c49-ea9a5cfd1dc7/graceful-termination/0.log"
Sep 18 00:20:54.759949 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:54.759895    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7lfv_508404ff-8f01-4fff-8f1f-cffeb8563192/kube-multus-additional-cni-plugins/0.log"
Sep 18 00:20:54.776687 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:54.776658    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7lfv_508404ff-8f01-4fff-8f1f-cffeb8563192/egress-router-binary-copy/0.log"
Sep 18 00:20:54.794780 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:54.794752    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7lfv_508404ff-8f01-4fff-8f1f-cffeb8563192/cni-plugins/0.log"
Sep 18 00:20:54.811717 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:54.811695    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7lfv_508404ff-8f01-4fff-8f1f-cffeb8563192/bond-cni-plugin/0.log"
Sep 18 00:20:54.828690 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:54.828662    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7lfv_508404ff-8f01-4fff-8f1f-cffeb8563192/routeoverride-cni/0.log"
Sep 18 00:20:54.856883 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:54.856858    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7lfv_508404ff-8f01-4fff-8f1f-cffeb8563192/whereabouts-cni-bincopy/0.log"
Sep 18 00:20:54.873586 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:54.873553    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-d7lfv_508404ff-8f01-4fff-8f1f-cffeb8563192/whereabouts-cni/0.log"
Sep 18 00:20:55.214199 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:55.214171    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-z62jl_fbf47e2b-0dfd-4a92-9042-8593f08a9a73/kube-multus/0.log"
Sep 18 00:20:55.233431 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:55.233399    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mpl4d_94abe8fb-5a88-4039-884a-1397dec13323/network-metrics-daemon/0.log"
Sep 18 00:20:55.250186 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:55.250155    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mpl4d_94abe8fb-5a88-4039-884a-1397dec13323/kube-rbac-proxy/0.log"
Sep 18 00:20:56.330916 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.330875    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/ovn-controller/0.log"
Sep 18 00:20:56.345954 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.345922    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/ovn-acl-logging/0.log"
Sep 18 00:20:56.367326 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.367296    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/kube-rbac-proxy-node/0.log"
Sep 18 00:20:56.385301 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.385258    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/kube-rbac-proxy-ovn-metrics/0.log"
Sep 18 00:20:56.400248 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.400211    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/northd/0.log"
Sep 18 00:20:56.417104 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.417074    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/nbdb/0.log"
Sep 18 00:20:56.434259 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.434229    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/sbdb/0.log"
Sep 18 00:20:56.495824 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:56.495791    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kwr95_2d03d11b-533f-4d38-9ca5-c431f7e34f21/ovnkube-controller/0.log"
Sep 18 00:20:57.751627 ip-10-0-132-155 kubenswrapper[2562]: I0918 00:20:57.751551    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ttv8n_c31f4265-c4c3-49a8-b13a-e84f31c0f6f2/network-check-target-container/0.log"